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July  24,  2002 

MEMORANDUM  FOR  UNDER  SECRETARY  OF  DEFENSE  FOR  PERSONNEL 

AND  READINESS 
DIRECTOR,  JOINT  STAFF 
DIRECTOR,  DEFENSE  INFORMATION  SYSTEMS 
AGENCY 


SUBJECT:  Report  on  the  Global  Command  and  Control  System  Readiness  Assessment 
System  Output  Tool  (Report  No.  D-2002-133) 


We  are  providing  this  report  for  review  and  comment.  We  considered 
management  comments  on  a  draft  of  this  report  when  preparing  the  final  report. 

DoD  Directive  7650.3  requires  that  all  recommendations  be  resolved  promptly. 
The  Joint  Staff  and  the  Defense  Information  Systems  Agency  were  responsive.  As  a 
result  of  management  comments,  we  added  Recommendation  1.  to  the  Under  Secretary 
of  Defense  for  Personnel  and  Readiness,  and  we  revised  recommendations  2.a.,  to  the 
Joint  Staff,  3. a.,  and  3.c.  to  the  Defense  Information  Systems  Agency.  Therefore,  we 
request  comments  from  the  Under  Secretary  on  Recommendation  1.,  and  from  the  Joint 
Staff  and  the  Defense  Information  Systems  Agency  on  revised  Recommendations  2.a. 
and  3.a.  by  September  24,  2002. 

If  possible,  please  provide  management  comments  in  electronic  format  (Adobe 
Acrobat  file  only).  Send  electronic  transmission  to  the  e-mail  address  cited  in  the  last 
paragraph  of  this  memorandum.  Copies  of  the  management  comments  must  contain  the 
actual  signature  of  the  authorizing  official.  We  cannot  accept  the  /Signed/  symbol  in 
place  of  the  actual  signature. 

We  appreciate  the  courtesies  extended  to  the  audit  staff.  For  additional 
information  on  this  report,  please  contact  Ms.  Evelyn  R.  Klemstine  at  (703)  604-9172 
(DSN  664-9172)  (eklemstine@dodig.osd.mil)  or  Mr.  Hugh  G.  Cherry  at 
(703)  604-9614  (DSN  664-9614)  (hgcherry@dodig.osd.mil).  See  Appendix  D  for  the 
report  distribution.  The  audit  members  are  listed  on  the  inside  back  cover  of  this 
r^rt. 


Acting  Assistant  Inspector  General 
for  Auditing 


Office  of  the  Inspector  General  of  the  Department  of  Defense 


Report  No.  D-2002-133  July  24,  2002 

(Project  No.  D2001LG-0101.01) 

Global  Command  and  Control  System  Readiness 
Assessment  System  Output  Tool 

Executive  Summary 


Who  Should  Read  This  Report  and  Why?  This  report  should  be  read  by  the  Global 
Command  and  Control  System  (GCCS)  requirements  developers,  program 
implementers,  functional  proponents,  and  readiness  users  who  rely  on  them.  The 
report  discusses  the  development  and  fielding  of  a  GCCS  candidate  solution  for  the 
readiness  functional  area. 

Background.  This  is  the  second  in  a  series  of  reports  the  Inspector  General  of  the 
Department  of  Defense  is  issuing  on  the  policies  and  procedures  that  govern  the  GCCS. 
The  GCCS  is  the  DoD  joint  command  and  control  system  designed  to  provide  the 
military  leadership  with  tools  to  plan  and  execute  worldwide  joint  military  operations. 
The  Readiness  Assessment  System  Output  Tool  is  a  GCCS  application  designed  to 
fulfill  both  Service  and  combatant  command  readiness  assessment  requirements  at  the 
tactical,  operational,  and  strategic  levels. 

Results.  The  Joint  Staff  and  the  Defense  Information  Systems  Agency  did  not  ensure 
that  development  of  the  Readiness  Assessment  System  Output  Tool  would  address  user 
needs.  Fielding  dates  for  the  Readiness  Assessment  System  Output  Tool  slipped  at  least 
5  years,  and  the  potential  users  were  not  involved  for  extended  periods.  In  addition, 
decisions  on  application  selection  were  not  fully  documented.  Readiness  users  and  the 
systems  developers  would  benefit  from  increased  user  involvement  in  requirement 
validations  and  testing  processes  as  well  as  more  effective  documentation  of  cost, 
feasibility  analyses,  goals,  and  decisions.  The  program  would  benefit  with 
development  of  concrete  and  quantifiable  performance  standards  that  can  be  accurately 
tested  as  well  as  a  formal  report  on  how  the  Readiness  Assessment  System  Output  Tool 
requirements  meet  performance  standards.  We  also  reviewed  the  management  control 
program  as  it  relates  to  the  development  and  fielding  of  GCCS  applications.  See  the 
Finding  section  for  details  on  the  audit  results  and  recommendations. 

Management  Comments  and  Audit  Response.  The  Director,  Joint  Staff,  the 
Director,  Defense  Information  Systems  Agency,  and  the  Deputy  Assistant  Secretary  of 
Defense  (Programs),  Office  of  the  Assistant  Secretary  of  Defense,  Command,  Control, 
Communications,  and  Intelligence  each  stated  that  planned  or  completed  actions 
sufficiently  addressed  the  recommendations.  After  review  of  feedback  from  the  user 
community  and  validation  of  requirements,  the  Readiness  Assessment  System  Executive 
Steering  Committee  directed  the  Defense  Information  Systems  Agency  to  reengineer  the 
Readiness  Assessment  System  Output  Tool.  The  Defense  Information  Systems  Agency 
installed  pre-release,  functioning  versions  of  the  Readiness  Assessment  Systems  Output 
Tool  for  users  to  evaluate  in  the  Joint  Staff,  four  Service  headquarters,  two  Service- 
Component  headquarters,  and  two  combatant  command  headquarters.  The  Joint  Staff 
required  user  representatives  to  participate  in  functional  user  and  operational  testing. 


The  users  were  eneouraged  to  submit  the  evaluations  at  any  time  during  the  review 
period.  The  Joint  Staff  plans  to  establish  an  additional  user  review  panel  that  will 
evaluate  the  requirements  not  ineorporated  into  the  Readiness  Assessment  System 
Output  Tool  schedule.  See  the  discussion  of  comments  in  the  recommendations  section 
and  Appendix  C.  We  consider  the  comments  received  to  be  responsive.  As  a  result  of 
comments  from  the  Joint  Staff,  we  added  a  recommendation  for  the  Under  Secretary  of 
Defense  for  Personnel  and  Readiness  to  publish  findings  from  research  on  readiness 
issues  for  the  Defense  Readiness  Reporting  System.  Further,  we  revised  the 
recommendations  for  the  Joint  Staff  and  the  Defense  Information  Systems  Agency  to 
recognize  progress  made  and  encourage  fumre  user  involvement.  We  request  the 
Under  Secretary  of  Defense  for  Personnel  and  Readiness  the  Joint  Staff  and  the  Defense 
Information  Systems  Agency  provide  comments  on  the  final  report  by 
September  24,  2002. 
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Background 


Global  Command  and  Control  System.  The  Global  Command  and  Control 
System  (GCCS)  is  the  DoD  joint  command  and  control  system  designed  to 
provide  the  leadership  from  the  National  Command  Authority^  through  the 
battlefield  commanders  with  tools  to  plan  and  execute  worldwide  joint  military 
operations.  GCCS  consists  of  interoperable  hardware,  software,  common 
procedures,  standards,  and  interfaces  that  comprise  an  “operational 
architecmre.”  GCCS  incorporates  force  planning,  readiness  assessment,  and 
simational  awareness^  applications  that  battlefield  commanders  require  to 
effectively  plan  and  execute  joint  military  operations. 

Global  Command  and  Control  Management  Structure.  Stakeholders  within 
the  Global  Command  and  Control  management  strucmre  participate  in  GCCS 
requirements  validation,  system  development,  and  system  implementation. 

Requirements  Validation.  The  Global  Command  and  Control  Review 
Board,  the  Joint  Staff,  GCCS  users,  and  working  groups  participate  in 
requirements  validation. 

Joint  Staff.  The  Joint  Staff,  as  the  project  sponsor  for  all  GCCS 
applications,  represents  the  needs  of  the  users.  Through  the  requirements 
validation  and  approval  process,  the  Joint  Staff  identifies  and  prioritizes 
functional  requirements  GCCS  must  satisfy. 

Global  Command  and  Control  Review  Board.  Representatives  from 
each  of  the  Joint  Staff  directorates.  Services,  combatant  commands,  and  GCCS 
working  groups  make  up  the  Global  Command  and  Control  Review  Board.  The 
Global  Command  and  Control  Review  Board  is  responsible  for  (1)  identifying, 
reviewing,  and  validating  requirements  added  to  GCCS;  (2)  approving  and 
prioritizing  requirements;  (3)  selecting  the  best  candidate  application  for 
integration  into  GCCS;  and  (4)  approving  policies  and  procedures  that  support 
joint  command  and  control  requirements. 

User  Community.  GCCS  users  within  the  combatant  commands. 
Services,  Defense  agencies,  or  the  Joint  Staff  may  identify  and  introduce  any 
GCCS  functional  requirements.  The  functional  users  participate  in  defining 
requirements  throughout  the  evaluation,  development,  and  fielding  of 
applications. 

GCCS  Working  Groups.  Working  groups  meet  regularly  and  allow 
operational  users  to  participate  in  developing  and  refining  input  for  GCCS 
strategies,  objectives,  requirements,  and  priorities.  GCCS  working  groups 


'The  National  Command  Authority  is  the  President  or  the  Seeretary  of  Defense  or  their  duly  deputized 
alternates  or  suceessors. 

^Situational  awareness  is  knowledge  of  friendly  and  adversary  capabilities  and  intentions  along  with  other 
relevant  information  that  enables  commanders  to  exercise  effective  command  and  control. 
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review  new  requirements,  validate  assigned  priorities,  evaluate  the  extent  to 
which  existing  applications  meet  new  requirements,  and  recommend  new 
requirements  to  the  Global  Command  and  Control  Review  Board. 

System  Development  and  Implementation.  The  Assistant  Secretary  of 
Defense  (Command,  Control,  Communications,  and  Intelligence)  (Assistant 
Secretary),  Defense  Information  Systems  Agency  (DISA),  and  Integrated 
Process  Teams  have  key  roles  in  GCCS  system  development  and 
implementation. 

Assistant  Secretary  of  Defense  (Command,  Control, 

Communications,  and  Intelligence).  The  Assistant  Secretary  provides 
functional  and  acquisition  control  over  GCCS  through  review  and  approval  of 
the  acquisition  strategy,  implementation  plans,  and  program  baseline  goals. 

Defense  Information  Systems  Agency.  As  the  centralized  program 
manager  for  GCCS,  DISA  directs  design,  development,  acquisition,  integration, 
testing,  fielding,  and  life-cycle  support  of  GCCS  applications.  The  program 
manager  is  also  responsible  for  developing  metrics  to  measure  program  progress 
and  effectiveness. 

Integrated  Process  Teams.  System  development  and  implementation 
concept  for  the  GCCS  rely  heavily  on  representatives  from  functional  disciplines 
that  work  together  in  Integrated  Process  Teams  to  build  a  successful  program 
and  enable  decision  makers  to  make  timely  decisions  that  will  facilitate  the 
acquisition  process. 

Readiness  Assessment  System.  In  1996,  the  Defense  Advanced  Research 
Projects  Agency  was  developing  the  Readiness  Assessment  System  (RAS)  as  an 
Advanced  Concept  Technology  Demonstration.^  DISA  and  the  Joint  Staff 
elected  to  add  RAS  to  GCCS.  The  addition  of  the  RAS  to  GCCS  would  provide 
a  comprehensive  readiness  reporting  system  and  offer  a  more  accurate  and 
timely  capability  to  measure  joint  readiness,  warfighting  deficiencies,  and  risk 
for  executing  wartime  missions.  Improving  asset  visibility  and  integrating 
leading  readiness  indicators  are  additional  goals  of  the  RAS. 

The  RAS  was  originally  intended  to  merge  four  independent  applications  into  a 
homogenous  system.  The  four  applications  were  the  Automated  Joint  Monthly 
Readiness  Review,  the  Joint  Exercise  Management  Program  III,  the  RAS  Input 
Tool,  and  the  RAS  Output  Tool."*  However,  the  Automated  Joint  Monthly 
Readiness  Review  has  since  been  eliminated,  and  the  Joint  Exercise 
Management  Program  III  was  replaced.  The  Marine  Corps  is  scheduled  to 


^Advanced  Concept  Technology  Demonstrations  are  intended  to  assess  the  military  utility  of  a  new 
eapability  and  to  conduet  an  assessment  at  a  seale  adequate  enough  to  clearly  establish  operational  utility 
and  system  integrity. 

'‘The  RAS  Input  Tool  and  the  RAS  Output  Tool  were  originally  referred  to  as  the  Global  Status  of 
Resourees  and  Training  System  (GSORTS)  Enhanced  Input  Tool  and  the  GSORTS  Enhaneed  Output 
Tool,  respeetively. 
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receive  the  RAS  Input  Tool  during  FY  2002,  and  the  Air  Force  is  scheduled  to 
receive  the  RAS  Input  Tool  during  the  third  quarter  of  FY  2003.  See 
Appendix  B  for  a  detailed  description  of  the  Automated  Joint  Monthly  Readiness 
Review,  the  Joint  Exercise  Management  Program  III,  and  the  RAS  Input  Tool. 

RAS  Output  Tool.  The  RAS  Output  Tool  was  designed  to  fulfill  both  Service 
and  combatant  command  readiness  assessment  requirements  at  the  tactical, 
operational,  and  strategic  levels.  As  a  part  of  GCCS,  the  RAS  Output  Tool  is 
intended  to  provide  near  real-time  access  to,  and  analysis  of,  readiness  data 
stored  in  the  GSORTS  database,^  Time-Phased  Force  and  Deployment  Data** 
stored  in  the  Joint  Operation  Planning  and  Execution  System’^  database,  and 
5  years  of  readiness  data.  By  employing  the  RAS  Output  Tool,  the  Services 
want  more  versatility  and  efficiency  for  reviewing  the  readiness  of  forces  and 
acquiring  insight  into  joint-  and  national-level  readiness  assessments.  The 
application  is  also  intended  to  allow  the  combatant  commands  to  assess  the 
readiness  of  forces  across  Service  lines  and  provide  readiness  assessment 
capabilities  required  to  plan  for  and  facilitate  timely  resource  allocation  and 
force  commitment  recommendations  to  the  National  Command  Authorities. 
DISA  planned  to  field  an  initial  version  of  the  RAS  Output  Tool,  followed  by 
four  Service-unique  versions  of  the  application. 

RAS  Development.  On  January  11,  1999,  DISA  awarded  a  contract 
(DCA100-99-C-4019)  to  A.B.  Eloyd  Enterprises,  Incorporated,  Alexandria, 
Virginia,  for  readiness  application  support,  maintenance,  and  enhancement. 
DISA  uses  the  contract  to  procure  RAS  Output  Tool  development  efforts. 

Objectives 


This  is  the  second  in  a  series  of  reports  that  the  Inspector  General  of  the 
Department  of  Defense  is  issuing  on  the  policies  and  procedures  that  govern  the 
GCCS.  The  overall  audit  objective  was  to  evaluate  the  joint  functionality, 
system  integration,  and  operations  of  GCCS.  The  specific  objective  for  this 
segment  of  the  audit  was  to  assess  management  and  oversight  of  the  RAS  Output 
Tool.  See  Appendix  A  for  a  discussion  of  the  audit  scope,  methodology, 
management  control  program,  and  prior  coverage. 


^The  GSORTS  database  is  a  read-only  look  into  the  Status  of  Resources  and  Training  System  database. 
Military  units,  some  DoD  organizations,  and  certain  foreign  and  international  organizations  involved  in 
operations  with  DoD  regularly  enter  data  on  the  status  of  unit  personnel,  training,  equipment  on  hand, 
and  equipment  serviceability. 

^Planners  use  Time-Phased  Force  and  Deployment  Data  (such  as  estimates  of  logistics  support 
requirements  and  designated  ports  for  loading  and  offloading)  to  identify  types  of  forces  and  actual  units 
required  to  support  an  operation  plan  and  establish  the  sequence  for  moving  forces  and  their  support 
(time  phasing)  into  an  area  of  operations. 

^The  Joint  Operation  Planning  and  Execution  System  is  a  mission  essential  GCCS  application  used  to 
conduct  joint  planning,  execution  (including  theater-level  nuclear  and  chemical  plans),  and  monitoring 
activities. 
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Readiness  Assessment  System  Output 
Tool  Development  and  Fielding 

DISA  and  the  Joint  Staff  did  not  provide  sufficient  assurance  that  the 
RAS  Output  Tool  would  satisfy  user  requirements.  In  addition,  DISA 
and  the  Joint  Staff  did  not  meet  RAS  Output  Tool  development  and 
fielding  milestones.  Those  conditions  occurred  because  DISA  and  the 
Joint  Staff  excluded  potential  users  for  extended  periods,  underestimated 
the  level  of  effort  to  develop  the  software,  did  not  fully  document 
critical  decisions  on  application  selection  and  development  estimates, 
and  did  not  establish  effective  performance  measurements  to  monitor 
development.  In  addition  turnover  of  key  personnel  and  failure  of  the 
application  to  meet  minimum  operational  standards  during  testing 
delayed  fielding.  As  a  result,  DoD  has: 

•  spent  $1.3  million  as  well  as  an  unknown  portion  of  an 
additional  $9.2  million  to  develop  the  RAS  Output  Tool;* 

•  estimated  that  an  additional  $5.6  million  will  be  required  to 
complete  development  and  field  the  application;  and 

•  projected  that  the  application  will  be  fielded  at  least  5  years 
late. 

In  addition,  delays  in  fielding  made  it  necessary  for  the  combatant 
commands  and  Services  to  either  develop  new  applications  or  modify 
existing  applications  that  would  satisfy  their  unique  requirements  for 
performing  readiness  analysis. 


Criteria 


Global  Command  and  Control  System  Development  Strategy.  In 

August  1993,  the  Assistant  Secretary  directed  that  GCCS  should  be  developed 
using  evolutionary  initiatives  designed  to  provide  to  users  rapid  and  efficient 
delivery  of  command  and  control  capabilities. 

In  June  1995,  the  Assistant  Secretary  further  redelegated  functional 
responsibilities  for  GCCS  to  ensure  more  effective  management  and  life-cycle 
support  for  the  system.  One  of  the  justifications  provided  for  use  of 
evolutionary  acquisition  initiatives  for  GCCS  was  that: 


*In  comments  provided  on  the  draft  report  DISA  stated  that  RAS  Output  Tool  expenditures  totaled 
approximately  $6.1  million  between  January  1998  and  Mareh  2002  of  whieh  $1.72  million  was  spent 
under  the  Defense  Advaneed  Research  Projects  Agency  management.  DISA  did  not  provide 
documentation  supporting  either  amount. 
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Traditional  acquisition  programs  in  the  past  have  not  been  able  to 
accommodate  rigid  milestones  programs,  contributing  to  stovepipe 
systems.  GCCS  .  .  .  will  allow  new  technology  and  leading  edge 
capabilities  to  be  integrated  much  faster  throughout  the  warfighter 
community. 

Requirements  Evaluation  Procedures.  Chairman  of  the  Joint  Chiefs  of  Staff 
Manual  6721.01,  “Global  Command  and  Control  System  (GCCS)  Functional 
Requirements  Evaluation  Procedures,”  March  15,  1997  (the  GCCS 
Requirements  Manual),  prescribes  the  specific  coordination  process  and  the 
responsibilities  for  assessing,  defining,  developing,  funding,  submitting, 
prioritizing,  and  validating  new  GCCS  requirements.  The  GCCS  Requirements 
Manual  establishes  the  following  as  the  standard  for  GCCS  program 
management  effectiveness. 

The  chief  consideration  of  the  development  process  is  to  accurately 
define  what  the  warfighter  needs,  find  the  best  solutions  government 
or  industry  has  to  offer,  and  make  a  decision  using  select  judgment 
criteria  to  implement  the  most  cost-effective  solution. 

The  requirements  and  responsibilities  within  the  GCCS  Requirements  Manual 
and  other  Chairman  of  the  Joint  Chiefs  of  Staff  guidance  are  projected  to  be 
revised  to  more  closely  align  with  the  approved  evolutionary  aequisition 
strategy.^ 

Evolutionary  Acquisition  Strategy.  On  September  18,  2000,  the  Assistant 
Secretary  approved  the  “Global  Command  and  Control  System  (GCCS) 
Evolutionary  Aequisition  Strategy,  Revision  2.2.”  The  purpose  of  the 
evolutionary  aequisition  strategy  is  to  align  the  GCCS  acquisition  strategy  with 
DoD  acquisition  reform  initiatives  and  oversight  policies  and  support  the  rapid 
development  and  implementation  of  the  GCCS. 

RAS  Business  Plan.  The  Seeretary  of  Defense,  in  the  “FY  1999-2003  Defense 
Planning  Guidanee,”^®  July  2,  1997,  requires  that  the  Chairman  of  the  Joint 
Chiefs  of  Staff  recommend  an  integrated  business  plan  for  establishing  a 
readiness  assessment  system  capable  of  addressing  the  full  spectrum  of  missions 
required  by  the  U.S.  Defense  strategy.  The  resulting  “RAS  Business  Plan,” 
June  1998,  outlines  the  development  plan  for  the  RAS.  The  preseribed 
development  plan  uses  an  ineremental  and  dynamie  approaeh  to  replaee  with 
newer  teehnology  the  antiquated  elements  of  GSORTS,  while  at  the  same  time 


^The  evolutionary  acquisition  strategy  further  details  the  roles  of  the  Joint  Staff  (J-3)  as  the  GCCS 
requirements  validator,  the  Assessment  Working  Integrated  Proeess  Team  in  researching  and  evaluating 
eandidate  solutions  to  validated  GCCS  requirements,  and  DISA  in  integrating  seleeted  eandidates 
applications  into  GCCS. 

“•The  Defense  Planning  Guidance,  signed  by  the  Secretary  of  Defense,  provides  firm  guidance  in  the 
form  of  goals,  priorities,  and  objectives  (including  fiscal  constraints)  that  the  Military  Departments  and 
Defense  agencies  use  to  develop  Program  Objective  Memorandums. 
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add  capabilities  the  changing  global  environment  requires.  The  RAS  Business 
Plan  explains  the  intent  of  each  RAS  application  and  outlines  development  and 
funding  timelines. 

Performance  Measurements. “  The  Clinger-Cohen  Act  of  1996  requires 
performance-based  management'^  and  results-based  management  to  reduce  the 
cost  of  programs  and  support  the  efficiency  and  effectiveness  of  information 
technology  investments.  The  Office  of  the  Secretary  of  Defense  Memorandum, 
“Requirements  for  Compliance  with  Reform  Legislation  for  Information 
Technology  (IT)  Acquisitions  (Including  National  Security  Systems),”  May  1, 
1997,  implements  the  requirements  of  the  Clinger-Cohen  Act  for  the  DoD.  The 
memorandum  implements  performance  measurements.  The  Clinger-Cohen  Act 
requires  Federal  organizations  to  measure  performance  and  devise  processes  that 
will  obtain  timely  information  on  the  progress  of  an  investment  in  an 
information  system.  The  Clinger-Cohen  Act  requires  that  organizations 
establish  performance  measurement  processes  that  address  independently 
verifiable  milestones  for  measuring  cost,  timeliness,  and  application  quality  and 
capability  of  specified  requirements.  Performance  measurement  processes 
maximize  the  value  as  well  as  assess  and  manage  the  risks  of  information 
technology  acquisitions. 

User  Requirements 


DISA  and  the  Joint  Staff  did  not  provide  sufficient  assurance  that  the  RAS 
Output  Tool  would  satisfy  user  requirements.  The  GCCS  Requirements  Manual 
specifies  that,  “user  participation  in  requirements  definition,  throughout 
evaluation,  development,  and  fielding  applications,  is  critical  to  the  successful 
implementation  of  GCCS  requirements.” 

User  Awareness  and  Concerns.  Potential  users  of  the  RAS  Output  Tool  did 
not  have  sufficient  assurance  that  the  RAS  Output  Tool  will  satisfy 
requirements.  To  assess  user  awareness  of  and  participation  in  the  RAS  Output 
Tool  requirement  identification  and  development,  we  surveyed  each  of  the  four 
Services  and  six'^  of  the  nine  combatant  commands.  As  of  December  2001,  the 
U.S.  European  Command,  the  U.S.  Pacific  Command,  the  U.S.  Special 
Operations  Command,  and  the  Air  Force  stated  they  had  limited  knowledge  of 


"Performance  measurement  is  an  assessment  of  the  effectiveness  and  efficiency  of  information 
technology  in  support  of  the  achievement  of  the  missions,  goals,  and  quantitative  objectives  of  an 
organization  through  the  application  of  outcome-based,  measurable,  and  quantifiable  criteria,  compared 
against  an  established  baseline  to  activities,  operations,  and  processes. 

'^Performance-based  management  is  a  systematic  approach  to  performance  improvement  through  an 
ongoing  process  of  establishing  strategic  performance  objectives;  measuring  performance;  collecting, 
analyzing,  reviewing,  and  reporting  performance  data;  and  using  that  data  to  drive  performance 
improvement. 

'^We  surveyed  the  following  combatant  commands  because  they  are  the  most  significant  potential  users  of 
the  RAS  Output  Tool:  U.S.  Central  Command,  U.S.  European  Command,  U.S.  Joint  Forces 
Command,  U.S.  Pacific  Command,  U.S.  Special  Operations  Command,  and  U.S.  Southern  Command. 
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the  RAS  Output  Tool.  In  addition,  the  U.S.  Central  Command,  the  U.S. 
Southern  Command,  and  the  Navy  all  stated  that  they  had  no  knowledge  of  the 
RAS  Output  Tool.  Several  of  the  users  voiced  concern  that  the  application  will 
not  provide  the  functionality  they  require.  For  instance,  the  U.S.  Joint  Forces 
Command  stated  that  the  RAS  Output  Tool  did  not  provide  Joint  Readiness 
Automated  Management  System  (JRAMS)’"^  functionality  and  performed 
significantly  slower  than  JRAMS.  Also,  Air  Force  officials  stated  that  they  had 
limited  communication  about  the  application  with  the  Joint  Staff. 

RAS  Business  Plan.  The  Joint  Staff  requested  that  the  Secretary  of  Defense 
and  the  Services  concur  with  the  proposals  for  developing  RAS  as  outlined  in 
the  RAS  Business  Plan.  Each  concurred;  however,  most  officials  voiced 
concern  regarding  the  need  to  define  requirements.  In  addition,  the  Army  and 
the  Air  Force  stated  that  the  requirements  needed  to  be  validated.  The  Joint 
Staff  did  not  request  concurrence  from  the  combatant  commands. 

Secretary  of  Defense  Comments.  The  Secretary  of  Defense  approved 
the  RAS  Business  Plan  on  September  15,  1999.  However,  the  approval  of  the 
plan  included  substantive  comments  that  should  be  included  when  the  plan  is 
revised.  The  Secretary  requested  revisions  that  would  accurately  reflect 
up-to-date  schedule  and  funding  information  as  well  as  emphasize  the  criticality 
of  involving  the  combatant  commands  and  other  users  in  the  development 
process: 


The  key  to  the  utility  and  effectiveness  of  the  RAS  is  the  involvement 
of  the  CINCs  [Commanders  in  Chief]  and  other  users  in  the 
development  process.  Early  involvement  will  improve  understanding 
of  the  potential  of  RAS  and  will  lead  to  the  identification  of  new 
requirements. 

Army  Comments.  The  Army  concurred  with  the  RAS  Business  Plan  on 
June  8,  1998.  Army  officials  did  not,  however,  believe  that  DISA  and  the  Joint 
Staff  performed  an  adequate  mission  needs  analysis  or  validated  the  requirement 
for  a  readiness  assessment  applieation  as  comprehensive  as  RAS.  To  determine 
the  precise  information  timeliness  requirements  at  eaeh  level  of  command  and  to 
target  information  to  user  needs,  the  Army  felt  that  a  more  thorough  mission 
needs  statement  would  be  required.  The  Army  also  believed  that  the  RAS 
Business  Plan  eost  estimates  were  unrealistie. 

Navy  Comments.  The  Navy  eoneurred  with  the  RAS  Business  Plan  on 
May  1,  1998.  Navy  officials  stated  that,  although  the  RAS  Business  Plan 
provided  a  reasonable  approach  to  developing  a  more  eomprehensive  and 
user-friendly  RAS,  additional  emphasis  should  be  plaeed  on  determining  user 
requirements  to  ensure  that  aeeurate  information  was  available  at  the  appropriate 
levels. 


‘'^The  Defense  Advanced  Research  Projects  Agency  developed  JRAMS  for  the  U.S.  Joint  Forces 
Command.  JRAMS  is  a  readiness  analysis  system  that  displays  joint  force  status  by  accessing, 
compiling,  and  displaying  information  from  disparate  data  sources  such  as  the  GSORTS  and  the  Joint 
Operation  Planning  and  Execution  System  databases. 


7 


Air  Force  Comments.  The  Air  Force  concurred  with  the  RAS  Business 
Plan  on  May  7,  1998.  Air  Force  officials  stated  that  the  plan,  however,  did  not 
detail  Service  requirements  for  RAS  fielding  and  that  a  coherent,  detailed  vision 
of  goals  was  needed.  Officials  recommended  that  the  Joint  Staff  and  DISA 
produce  a  statement  of  need  and  a  requirements  document  with  specific,  detailed 
tasks  and  goals. 

Marine  Corps  Comments.  The  Marine  Corps  concurred  with  the  RAS 
Business  Plan  on  April  29,  1998.  Officials  stated  that,  while  readiness 
operational  requirements  were  derived  in  response  to  areas  important  to  the 
Joint  Monthly  Readiness  Review,  the  Marine  Corps  believed,  given  the  level  of 
funding  applied  to  the  RAS  effort,  that  RAS  oversight  should  enable  a  review  of 
future  RAS  requirements  and  a  review  of  progress. 

As  of  June  2002,  DISA  and  the  Joint  Staff  had  not  updated  the  RAS  Business 
Plan  and  could  not  document  that  they  both  had  acted  on  the  direction  the 
Secretary  of  Defense  and  the  Services  provided.  In  addition,  DISA  and  the 
Joint  Staff  had  not  requested  input  or  concurrence  from  the  combatant 
commands. 

Testing.  DISA  and  the  Joint  Staff  did  not  involve  potential  users  in  application 
testing  for  extended  periods.  With  the  exception  of  U.S.  Joint  Forces 
Command,  the  combatant  commands  and  the  Services  had  not  participated  in 
RAS  Output  Tool  testing.  U.S.  Joint  Forces  Command  readiness  analysts  stated 
that  the  functional  qualification  tests  in  which  they  participated  were  not 
adequate  because  the  tests  did  not  provide  a  comprehensive  assessment  of  the 
functionality  that  the  application  would  offer.  Officials  stated  that  the  tests  did 
not  help  the  developers  or  the  users  discern  whether  the  RAS  Output  Tool 
would  be  “operationally  competent.”  Further,  U.S.  Joint  Forces  Command 
officials  stated  that  the  speed  of  the  application  and  the  quality  of  the  graphics 
were  not  adequate.  During  calendar  years  2000  and  2001  DISA  provided 
limited  hands-on  testing  experience,  which  has  diminished  the  potential  that 
users  will  gain  confidence  that  the  RAS  Output  Tool  will  satisfy  their  needs. 

The  Joint  Staff  and  DISA  did  not  fully  document  critical  decisions  on 
application  selection.  In  addition,  an  adequate  mechanism  for  measuring 
performance  was  not  established. 

Support  for  Software  Transition.  Joint  Staff  and  DISA  officials  did  not 
document  how  they  concluded  that  the  RAS  Output  Tool  could  be  efficiently  and 
effectively  fielded.  In  1998,  DISA  transitioned  the  RAS  Output  Tool  software 
from  the  Defense  Advanced  Research  Program  Agency  and  began  funding 
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development  of  the  application  in  GCCS.  Joint  Staff  officials  assessed  the  status 
of  the  application  upon  transition  from  the  Defense  Advanced  Research  Program 
Agency.  Officials  concluded  that: 

•  only  moderate  hardware  and  software  modifications  would  be  required  to 
prepare  the  application  for  fielding,  and 

•  the  RAS  Output  Tool  could  be  fielded  within  18  months  at  a  cost  of  less 
than  $1  million. 

Officials  did  not  maintain  documentation  on  development  and  fielding  estimates. 

Search  for  Candidate  Solutions.  DISA  could  not  document  that  they  either 
searched  for  or  analyzed  alternatives  that  would  satisfy  RAS  Output  Tool 
requirements.  The  GCCS  Requirements  Manual  specifies  that  a  search  for 
candidate  solutions  for  a  new  joint  requirement  should  occur  throughout  the 
DoD.  The  purpose  of  the  search  was  to  determine  the  extent  to  which  existing 
applications  might  economically  provide  the  necessary  functions  in  a  timely 
manner.  According  to  the  GCCS  Requirements  Manual,  GCCS  working  groups 
should  solicit  input  on  candidate  applications  that  may  satisfy  new  user 
requirements  from  the  users.  Once  the  candidate  applications  are  identified, 
DISA  should  document  any  cost  benefit  analyses  of  technical  solutions  and 
recommend  the  best  technical  solution  for  GCCS  implementation  that  satisfies 
the  requirement,  is  cost  effective,  and  is  feasible  to  implement. 

Analysis  of  Alternatives.  Existing  Service  or  combatant  command  tools 
might  have  met  RAS  Output  Tool  requirements,  but  the  Joint  Staff  and  DISA 
could  not  document  that  they  conducted  an  appropriate  search  of  candidate 
applications  for  the  RAS  Output  Tool.  DISA  officials  stated  that  they  thought 
seven  systems  were  considered  as  alternative  solutions  to  satisfy  RAS  Output 
Tool  requirements.  However,  neither  Joint  Staff  nor  DISA  officials  maintained 
any  records  that  confirmed  a  search  took  place.  Officials  did  not  maintain  any 
documentation  that  would  support  their  conclusion  that  no  other  system  would 
meet  the  requirement. 

Joint  Readiness  Automated  Management  System.  The  Joint  Staff  could 
not  document  that  they  analyzed  JRAMS  as  a  solution  that  would  meet  RAS 
Output  Tool  requirements.  The  RAS  Business  Plan  specifies  that  the  RAS 
Output  Tool  include  the  technical  infrastructure  and  the  viewing  capabilities  of 
JRAMS.  JRAMS  is  an  operationally  mature  program  that  the  U.S.  Joint  Forces 
Command,  the  U.S.  European  Command,  and  the  U.S.  Pacific  Command  have 
used.  The  functionality  of  JRAMS  mirrors  that  of  the  RAS  Output  Tool,  but 
without  the  historic  and  trend  capability  and  certain  Service-unique 
characteristics.  The  combatant  commands,  the  Joint  Staff,  and  DISA  provided 
numerous  laudatory  comments  about  JRAMS.  The  Joint  Staff  documented  that 
JRAMS  was  a  possible  alternative  to  the  RAS  Output  Tool,  further  stating  that  it 
was  the  best  application  that  would  meet  the  requirements  and  that  JRAMS  was 
worthy  of  analysis.  Both  the  Joint  Staff  and  DISA  officials  were  unable. 
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however,  to  provide  a  cost  benefit  analysis  or  any  analysis  on  the  technical 
feasibility  of  implementing  JRAMS  as  a  solution  to  satisfy  RAS  Output  Tool 
requirements. 

Performance  Measurements.  DISA  and  the  Joint  Staff  did  not  develop  an 
adequate  mechanism  that  would  measure  program  performance  against 
standards.  DISA  and  Joint  Staff  officials  did  not  develop  performance  standards 
for  GCCS  that  could  be  applied  to  the  RAS  Output  Tool.  Specifically,  DISA 
and  the  Joint  Staff  did  not  establish  a  performance  standard  that  tied  the  RAS 
Output  Tool  to  a  set  of  clearly  understood,  quantifiable  performance  measures 
that  would  objectively  assess  the  reliability,  technical  quality,  cost,  and  schedule 
baselines  for  the  application.  Without  performance  measurements,  officials 
could  not  adequately  evaluate  the  ability  of  the  application  to  meet  user  needs. 

Fielding  of  the  RAS  Output  Tool 


DISA  and  the  Joint  Staff  did  not  meet  development  and  fielding  milestones  for 
the  RAS  Output  Tool.  The  projected  fielding  date  for  the  RAS  Output  Tool 
slipped  44  months— from  February  2000  to  November  2003.  Difficulties  with 
the  software,  turnover  of  key  personnel,  and  the  RAS  Output  Tool  failure  to 
meet  minimum  operational  standards  caused  numerous  fielding  delays.  The 
table  below  depicts  the  delays  in  the  projected  RAS  Output  Tool  fielding  dates, 


RAS  Output  Tool  Fielding  Delays 

Documentation 

Projected  Completion 
Date 

Time  Past  Initial 
Fielding  Date 

RAS  Business  Plan 

June  1998 

February  28,  2000 

N/A 

In  Progress  Review 
November  1999 

December  15,  2000 

1 1  months 

In  Progress  Review 
November  2000 

September  18,  2001 

1  year,  8  months 

Working  Group 
Meetings 
February  2001 

October  23,  2001 

1  year,  9  months 

Executive  Steering 
Committee 

July  2001 

November  1,  2003 

3  years,  8  months 
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Fielding  Schedule.  The  RAS  Business  Plan  specifies  that  DoD  field  no  later 
than  February  28,  2000,  an  initial  version,  plus  four  Service-unique  versions  of 
the  RAS  Output  Tool.  As  of  January  31,  2002,  DISA  delayed  fielding  the 
initial  version  of  the  RAS  Output  Tool  four  times  and  projected  that  the  initial 
version  would  not  be  fielded  until  November  2003.  DISA  did  not  expect  to 
complete  fielding  of  the  Service-unique  versions  of  the  RAS  Output  Tool  until  at 
least  FY  2006. 

In-Progress  Reviews.  During  the  RAS  In-Progress  Review  held  on 
November  24,  1999,  DISA  officials  changed  the  initial  fielding  date  of  the  RAS 
Output  Tool  to  December  15,  2000,  because  of  “difficulties  with  the  initial 
software  delivery  being  nonfunctional.”  MITRE,  Incorporated,  suggested  in  an 
independent  assessment  of  the  RAS  Output  Tool  completed  in  November  1999 
that  the  DoD  estimates  of  costs,  resources,  and  time  were  “aggressive.”  At  a 
November  22,  2000,  RAS  In-Progress  Review,  DISA  again  rescheduled  to 
September  18,  2001,  the  initial  fielding  date  for  the  RAS  Output  Tool.  DISA 
cited  significant  changes  in  key  personnel  as  the  reason  for  the  9-month  schedule 
slippage. 

GCCS  Fielding  Schedule.  In  February  2001,  DISA  chose  to  postpone 
fielding  the  RAS  Output  Tool  and  release  the  application  as  part  of  GCCS 
Version  3.4.  DISA  scheduled  fielding  of  GCCS  Version  3.4  and  an  initial 
version  of  the  RAS  Output  Tool  for  October  23,  2001. 

Redesign.  After  a  Joint  Staff  review  of  the  detailed  readiness 
requirements,  the  RAS  Executive  Steering  Committee‘s  decided  on  July  25, 
2001,  to  delay  fielding  the  RAS  Output  Tool  until  November  2003.  Joint  Staff 
officials  explained  that  revised  performance  requirements  for  the  RAS  Output 
Tool  required  a  redesign  of  the  basic  readiness  architecture.  The  Joint  Staff 
believed  that  unless  design  flaws  were  rectified,  the  RAS  Output  Tool  would 
continue  failing  to  meet  minimum  operational  standards.  The  projected  fielding 
date  of  November  2003  coincides  with  the  fielding  date  of  GCCS  Version  4.2. 

Future  Versions  of  the  RAS  Output  Tool.  DISA  does  not  expect  to 
complete  fielding  of  the  Service-unique  versions  of  the  RAS  Output  Tool  until  at 
least  FY  2006.  The  RAS  Business  Plan  specifies  that  all  of  the  versions  of  the 
RAS  Output  Tool  would  be  fielded  by  February  28,  2000.  However,  the  four 
Service-unique  versions  of  the  application  were  postponed  as  a  result  of  delays 
in  fielding  the  initial  version  of  the  application. 

RAS  Output  Tool  Costs 


DISA  has  spent  $1.3  million  to  develop  the  RAS  Output  Tool.  DISA  has  also 
spent  an  unallocated  portion  of  an  additional  $9.2  million  to  support  the  RAS 


^^The  RAS  Executive  Steering  Committee  is  an  informal  process  action  team  created  to  direct  the 
development  of  GCCS  applications.  The  Committee  is  made  up  of  senior  officials  from  the 
Directorates  for  Operations  (J-3)  and  Command,  Control  Communications  and  Computers  (J-6),  Joint 
Chiefs  of  Staff,  representatives  of  DISA  headquarters  staff,  and  GCCS  Program  Management  Office. 
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Output  Tool.  DISA  estimated  that  another  $5.6  million  is  required  to  develop 
and  field  the  RAS  Output  Tool.  With  the  fielding  of  the  RAS  Output  Tool 
5  years  behind  schedule,  the  combatant  commands  and  Services  have  already 
begun  modifying  existing  applications  or  have  already  begun  developing  new 
applications  that  will  satisfy  their  unique  requirements  for  performing  readiness 
analysis. 

Quantifying  RAS  Output  Tool  Funds.  Between  January  1999  and 
December  2001,  DISA  spent  more  than  $10.5  million  for  the  RAS  development 
costs.  Figure  1  illustrates  expenditures  for  readiness  application  support  from 
January  1999  through  September  2001. 


$9,176,626 


I  *An  unknown  portion  was  applied  to  the  RAS  Output  Tool. _ 

Figure  1.  Readiness  Application  Support 

RAS  Output  Tool  Specific  Funds.  The  $1,326,013  that  DISA  spent  on 
development  of  the  RAS  Output  Tool  covered  technical  documents, 
enhancements,  meeting  support,  and  system  redesign.  During  calendar  year 
2000,  the  RAS  Output  Tool  continued  to  share  enhancement  costs  with  other 
readiness  applications.  The  RAS  Output  Tool  incurred  $866,763  of 
enhancement  costs  during  calendar  year  2001,  as  illustrated  in  Figure  2.  DISA 
also  disbursed  during  calendar  year  2001  $158,919  developing  technical 
documents  for  the  RAS  Output  Tool  and  $290,000  for  a  critical  redesign  of  the 
application. 
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■  Technical  Documents 
H  Enhancements 

□  Meeting  Support 

■  System  Redesign 


Figure  2.  RAS  Output  Tool  Development  Expenses 


Projected  Future  Costs.  As  of  November  22,  2000,  DISA  projected  that  an 
additional  $5.6  million  over  5  years  would  be  required  to  field  the  four  Service- 
unique  versions  of  the  RAS  Output  Tool  after  the  initial  version  of  the 
application  was  fielded.  The  Army  version  is  scheduled  for  fielding  in  October 
2002,  followed  by  the  Navy  version  in  December  2003,  the  Air  Force  version  in 
January  2005,  and  the  Marine  Corps  version  in  December  2005.  Figure  3 
illustrates  the  projected  costs  for  Service -unique  versions  of  the  RAS  Output 
Tool. 
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$1.86 


$1.32  $1.38 
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Service-Unique  Versions 


Figure  3.  Projected  Development  Costs  for  Service-Unique  Versions 
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Additional  Readiness  Assessment  Tools.  During  the  5  years  that  the  RAS 
Output  Tool  has  been  in  development,  the  U.S.  Joint  Forces  Command,  the 
Army,  and  the  Air  Force  have  either  developed  new  applications  or  modified 
existing  applications  to  help  satisfy  unique  readiness  analysis  requirements. 
Because  of  the  absence  of  a  GCCS  baseline  application  for  conducting  readiness 
analyses,  joint  readiness  action  officers  use  various  Service-  and  combatant 
command-unique  applications  to  extract  data  they  need  from  the  GSORTS  as 
well  as  the  Joint  Operation  and  Planning  Execution  System  databases  to  prepare 
and  present  readiness  analyses.  Continued  delays  in  fielding  the  RAS  Output 
Tool  have  already  required  the  users  to  perform  readiness  analysis  of  complex 
issues  without  the  benefit  of  a  mission  essential,  up-to-date  automation  product. 
Combatant  commands  and  the  Services  have  also  continued  to  invest  in 
upgrading  existing  systems  or  developing  new  readiness  applications  that  will 
meet  their  needs.  For  instance,  the  U.S.  Joint  Forces  Command  continues  to 
maintain  the  JRAMS  and  the  Army  continues  to  support  the  Army  Readiness 
Management  System.  In  addition,  the  Air  Force  has  begun  efforts  to  develop  a 
new  readiness  analysis  system  that  includes  predictive  capabilities. 

Conclusion 


The  DoD  has  spent  more  than  5  years  and  millions  of  dollars  attempting  to 
develop  and  field  a  readiness  analysis  application.  Also,  DISA  does  not  appear 
to  have  thoroughly  investigated  the  costs  and  feasibility  of  fielding  the  RAS 
Output  Tool  and  does  not  appear  to  be  testing  operational  performance 
requirements  that  agree  with  user  requirements.  Consequently,  the  program  has 
encountered  numerous  fielding  delays  and  DISA  started  a  redesign  effort  to 
make  the  application  operational.  Furthermore,  the  RAS  Output  Tool  may  not 
satisfy  readiness  analysis  requirements  for  the  user.  In  addition,  DoD  did  not 
establish  performance  measures  that  would  facilitate  effective  program 
management  and  oversight  and  help  ensure  DISA  fields  a  readiness  analysis 
application. 

Management  Comments  on  the  Finding  and  Audit  Response 


Summaries  of  management  comments  on  the  finding  and  our  audit  response  are 
in  Appendix  C. 

Recommendations,  Management  Comments,  and  Audit 
Response 


Added,  Deleted,  Renumbered,  and  Revised  Recommendations.  As  a  result 
of  management  comments,  we  added  Recommendation  1.  to  share  with  the 
entire  DoD  readiness  community  readiness  information  that  is  beneficial.  We 
deleted  draft  Recommendation  2.e.,  and  renumbered  all  recommendations  to 
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account  for  the  changes.  We  revised  draft  Recommendations  La.  and  2. a.  to 
account  for  recent  strides  in  the  development  of  the  RAS  Output  Tool  and  draft 
Recommendation  2.c.  to  improve  clarity. 

1.  We  recommend  that  the  Under  Secretary  of  Defense  for  Personnel  and 
Readiness  publish  findings  from  research  on  readiness  issues  for  the 
Defense  Readiness  Reporting  System.  The  Joint  Staff  suggested  that  the 
results  from  that  research  would  benefit  the  DoD  readiness  community. 

2.  We  recommend  that  the  Director  for  Operations,  Joint  Staff: 

a.  Proceed  with  planned  testing  in  June  2002  of  Readiness 
Assessment  System  Output  Tool  Version  4.0  requirements,  request  users 
attest  that  testing  results  met  their  requirements  for  fielding  an  application, 
and  revalidate  the  remaining  requirements  with  current  users  within  the 
combatant  commands.  Services,  and  Defense  agencies.  If  users  do  not 
attest  that  the  tool  is  adequate  for  fielding,  then  revalidate  prior  to 
proceeding  with  any  further  development  the  mission  need  as  well  as  all  of 
the  requirements. 

Joint  Staff  Comments.  The  Joint  Staff  concurred  with  the  intent  of  the 
recommendation  but  requested  a  modification  in  its  execution.  The  Joint  Staff 
will  invite  combatant  commands,  Combat  Support  Agencies,  and  the  Services  to 
review  the  more  than  290  user  requirements  and  any  other  requirements  that 
will  be  collected  and  earmarked  for  RAS  Output  Tool  versions  beyond  4.0  after 
the  June  2002  RAS  Output  Tool  test  is  completed.  DISA  and  the  Joint  Staff  will 
then  be  able  to  publish  the  results  of  the  test  and  ask  the  users  to  review  and 
comment  on  the  remaining  user  requirements  that  could  potentially  go  into 
future  RAS  Output  Tool  software.  The  395  requirements  scheduled  for  release 
with  GCCS  Version  4.0  must  not  change.  As  of  April  2002,  DISA  has  satisfied 
370  of  the  395  requirements  scheduled  for  inclusion  in  GCCS  Version  4.0. 
Those  requirements,  however,  cannot  change  without  either  unacceptably 
impeding  or  delaying  the  development  process. 

DISA  Comments.  DISA  stated  that  the  Joint  Staff  had  reviewed  the 
requirements  and  briefed  the  results  to  the  RAS  Executive  Steering  Committee 
in  July  2001 .  The  Joint  Staff  and  DISA  stated  they  plan  to  review  the  user 
requirements  list  again  in  May  2002  in  preparation  of  RAS  Output  Tool 
functional  user  testing  scheduled  for  June  2002.  The  Joint  Staff  plans  to  include 
combatant  command.  Combat  Support  Agency,  and  readiness  divisions  of  the 
Services  in  the  planned  review  of  user  requirements  slated  for  versions  beyond 
4.0  of  the  RAS  Output  Tool  as  well  as  any  additional  requirements  collected. 

Deputy  Assistant  Secretary  Comments.  The  Deputy  Assistant  Secretary  of 
Defense  (Programs)  stated  that  the  Joint  Staff  Directorate  for  Operations  (J-3) 
had  already  revalidated  in  May  2001  the  user  requirements  for  the  RAS  Output 
Tool.  In  addition,  the  Joint  Staff  and  DISA  plan  to  review  the  user  requirement 
list  in  May  2002  to  prepare  for  functional  user  testing  of  the  RAS  Output  Tool 
scheduled  for  June  2002. 
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Further,  the  Joint  Staff  plan  to  review  user  requirements  as  well  as  any 
additional  requirements  collected  for  RAS  Output  Tool  versions  beyond  4.0  with 
all  users. 

Audit  Response.  The  Joint  Staff  comments  are  fully  responsive.  As  a  result  of 
those  comments,  we  revised  the  recommendation  to  require  revalidation  of  the 
RAS  Output  Tool  requirements  only  if  the  Version  4.0  testing  proves  the  system 
is  unsuitable  for  fielding.  We  request  additional  comments  from  the  Joint  Staff 
on  the  revised  recommendation. 

b.  For  future  GCCS  readiness  applications,  search  Government  and 
commercial  sources  for  applications  that  will  satisfy  requirements  and 
document  the  results. 

Joint  Staff  Comments.  The  Joint  Staff  did  not  concur  with  the 
recommendation  because  a  Joint  Staff  effort  would  be  redundant  and  a  waste  of 
time  and  money.  The  Office  of  the  Under  Secretary  of  Defense  for  Personnel 
and  Readiness  (Under  Secretary  of  Defense)  looked  at  readiness  applications 
throughout  DoD  to  conceptualize  and  develop  an  improved  readiness  reporting 
system  for  DoD.  Because  the  DoD  readiness  community  would  benefit,  the 
Joint  Staff  suggested  that  the  report  should  recommend  that  the  Office  of  the 
Secretary  of  Defense  publish  findings.  The  Joint  Staff  will  make  sure  future 
studies  of  GCCS  candidate  applications  are  thoroughly  documented  and 
archived. 

Deputy  Assistant  Secretary  Comments.  The  Deputy  Assistant  Secretary  stated 
that  the  Under  Secretary  of  Defense,  DISA,  and  the  Joint  Staff  readiness 
reporting  initiatives  fulfill  this  requirement.  The  Joint  Staff  is  involved  in  a 
similar  process  through  the  development  of  an  operational  requirements 
document  for  GCCS. 

DISA  Comments.  DISA  stated  that  Under  Secretary  of  Defense  initiatives  to 
conceptualize  and  develop  an  improved  readiness  reporting  system  for  DoD  are 
fulfilling  this  requirement.  The  Joint  Staff  and  DISA  are  involved  in  the  effort. 
The  Joint  Staff  is  also  involved  in  a  similar  process  as  part  of  the  GCCS 
operational  requirements  document  development.  Additional  efforts  in  this 
direction  would  be  redundant. 

Audit  Response.  Although  the  Joint  Staff  nonconcurred  with  the 
recommendation,  the  Joint  Staff  agreed  to  ensure  that  future  studies  of  GCCS 
candidate  applications  are  thoroughly  documented  and  archived.  The  action 
satisfies  the  intent  of  the  recommendation. 

c.  Develop  performance  standards  that,  as  a  minimum,  provide  a 
way  of  assessing  the  performance,  cost,  schedule,  technical  quality  and 
reliability,  and  user  satisfaction  and  formally  report  on  the  progress  of  the 
Readiness  Assessment  System  Output  Tool  in  meeting  those  standards. 

Joint  Staff  Comments.  The  Joint  Staff  concurred  with  the  recommendation. 
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Deputy  Assistant  Secretary  Comments.  The  Deputy  Assistant  Secretary  stated 
that  the  recommendation  tasks  the  Joint  Staff  to  perform  activities  outside  their 
established  role.  The  GCCS  program  manager  is  responsible  for  managing  cost, 
schedule  and  performance,  and  reporting  in  accordance  with  DoD 
Regulation  5000. 2-R,  “Mandatory  Procedures  For  Major  Defense  Acquisition 
Programs  and  Major  Automated  Information  System  Acquisition  Programs,” 
April  5,  2002.  The  GCCS  program  manager  is  also  responsible  for  developing 
a  plan  that  integrates  test,  accreditation,  and  fielding  the  RAS  Output  Tool  as 
part  of  GCCS.  Office  of  the  Secretary  of  Defense  Director  for  Program 
Assessment  and  Evaluation  independently  assesses  the  program  costs  through 
review  of  the  program’s  economic  analysis.  The  Assistant  Secretary  provides 
oversight  of  the  program  while  the  Joint  Staff  and  users  provide  assessment  of 
the  functionality  that  is  delivered  in  the  RAS  Output  Tool. 

DISA  Comments.  DISA  stated  that  the  Joint  Staff  identifies,  validates,  and 
prioritizes  GCCS  requirements.  The  Joint  Staff  also  provides  to  DISA  objective 
and  threshold  functional  performance  parameters.  The  DISA  GCCS  program 
manager  plans  for  the  development,  integration,  testing,  accreditation,  and 
fielding  of  the  RAS  Output  Tool  as  part  of  the  larger  GCCS  program.  The  Joint 
Staff  represents  the  users  to  assess  how  well  the  developed  functions  meet  the 
validated  requirements  as  well  as  satisfaction  with  developed  functionality. 
However,  the  program  manager  assesses  cost,  schedule,  and  performance  and 
then  reports  the  overall  GCCS  status  by  way  of  quarterly  Defense  Acquisition 
Executive  Summaries  the  acquisition  oversight  authority. 

Audit  Response.  The  Joint  Staff  comments  are  responsive.  We  recognize  that 
the  Joint  Staff  and  the  GCCS  Program  Manager  have  differing  responsibilities  in 
requirements  development,  program  management  and  assessment.  If  each 
diligently  performs  their  delegated  responsibilities  as  outlined  in  the  Deputy 
Assistant  Secretary  and  DISA  comments,  we  consider  the  responses  meet  the 
intent  of  the  recommendation. 

3.  We  recommend  that  the  Director,  Defense  Information  Systems  Agency: 

a.  If  testing  in  June  2002  does  not  meet  minimum  needs  of  the  users 
for  Relding  as  the  users  stated,  then  pause  development  of  the  Readiness 
Assessment  System  Output  Tool  until  the  questions  with  requirements 
development,  user  satisfaction,  program  documentation,  and  performance 
measurement  have  been  fully  addressed. 

DISA  Comments.  DISA  did  not  concur  with  the  recommendation  because  a 
pause  in  development  would  duplicate  the  RAS  Executive  Steering  Committee 
review  that  was  performed  during  the  spring  and  early  summer  2001.  Based  on 
a  review  that  validation  of  requirements,  functional  qualification  tests,  and  user 
feedback,  the  RAS  Executive  Steering  Committee  directed  in  July  2001  that 
DISA  reengineer  the  RAS  Output  Tool.  Since  October  2001,  DISA  has 
installed  “pre-release,  functioning  versions  of  the  RAS”  output  tool  within  the 
Joint  Staff,  headquarters  of  the  four  Services,  two  Service  component 
headquarters,  and  two  combatant  command  headquarters.  The  first  test 
involving  users  of  an  operational  system  from  combatant  commands  and  the 
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Services  is  scheduled  for  June  2002.  Another  development  pause  would 
exacerbate  the  condition  that  caused  scheduling  delays.  Delays  at  this  stage  for 
staff  analysis  would  be  inappropriate,  costly,  and  counterproductive  for 
delivering  the  capability  to  the  user. 

Deputy  Assistant  Secretary  Comments.  The  Deputy  Assistant  Secretary  stated 
that  delays  for  staff  analysis  at  this  stage  would  be  inappropriate,  costly,  and 
counterproductive  for  delivering  the  capability  to  the  user.  The  Joint  Staff  and 
DISA  have  taken  actions  that  will  ensure  delivery  of  the  required  RAS  Output 
Tool  capability.  Based  on  a  review  and  validation  of  requirements,  a  functional 
qualification  test,  and  feedback  from  the  user,  the  RAS  Executive  Steering 
Committee  directed  in  July  2001  that  DISA  reengineer  the  RAS  Output  Tool. 

In  October  2001,  DISA  began  installing  pre-release  and  functioning  versions  of 
the  RAS  Output  Tool  within  the  Joint  Staff,  headquarters  of  the  four  Services, 
two  of  the  Service  component  headquarters,  and  two  combatant  command 
headquarters.  The  first  tests  of  an  operational  system  were  scheduled  for  June 
2002.  Users  will  be  drawn  from  the  combatant  commands  and  the  Services. 
Another  development  pause  would  exacerbate  the  condition  that  caused 
scheduling  delays.  Previous  delays  were  the  result  of  failures  in  the  software  to 
meet  user  requirements. 

Joint  Staff  Comments.  The  Joint  Staff  recommended  against  this  course  of 
action.  The  rate  of  the  developmental  progress  for  the  RAS  Output  Tool  has 
never  been  greater  than  now.  In  October  2001,  DISA  began  installing  pre¬ 
release  and  functioning  versions  of  the  RAS  Output  Tool  within  the  Joint  Staff, 
headquarters  of  the  four  Services,  two  of  the  Service  Component  headquarters, 
and  two  combatant  command  headquarters.  An  additional  prototype  will  soon 
be  installed  at  another  combatant  command.  Initial  response  to  the  prototype 
was  favorable.  The  first  user  test  of  a  production  system  was  scheduled  for 
June  2002.  Users  will  be  drawn  from  the  readiness  community,  combatant 
commands,  the  Services,  and  at  least  one  Combat  Support  Agency.  A  pause  in 
the  schedule  would  exacerbate  the  condition  that  caused  scheduling  delays. 
Previous  delays  were  the  result  of  failures  in  the  software.  Delays  for  staff 
analysis  at  this  stage  would  be  inappropriate.  The  recommendations  the 
Inspector  General  of  the  Department  of  Defense  described  can  be  accomplished 
in  a  parallel  effort. 

Audit  Response.  DISA  nonconcurred  with  the  recommendation  because  a 
pause  in  the  program  would  be  detrimental.  We  acknowledge  the  advancement 
in  RAS  Output  Tool  development  since  the  beginning  of  our  audit,  and  also 
understand  DISA,  Deputy  Assistant  Secretary,  and  Joint  Staff  concerns  about 
further  delay  for  a  system  that  is  scheduled  for  release  within  the  next  few 
months.  We  modified  the  recommendation  and  request  comments  to  the  revised 
recommendation  from  DISA. 

b.  Develop  and  document  cost  and  feasibility  analyses  of  alternative 
solutions  proposed  to  satisfy  Joint  Staff  identified  requirements. 

DISA  Comments.  DISA  concurred  with  the  recommendation.  DISA  has  a 
documented  process  in  the  GCCS  evolutionary  acquisition  strategy  for 
developing  cost  and  feasibility  analyses  of  candidate  solutions.  The  GCCS 


18 


Block  IV  Implementation  Plan  will  include  the  capabilities  that  successfully  pass 
cost  and  feasibility  analyses  and  show  how  the  capabilities  are  integrated  into 
GCCS.  DISA  plans  to  complete  the  Block  IV  Implementation  Plan,  provide  it 
to  the  GCCS  Integrating  Integrated  Product  Team,  and  submit  it  for  GCCS 
Milestone  Decision  Authority  approval  in  the  fourth  quarter  of  FY  2002. 

Deputy  Assistant  Secretary  Comments.  The  Deputy  Assistant  Secretary  stated 
that  the  Joint  Staff  is  responsible  for  developing  GCCS  requirements,  and  not 
identifying  solutions.  The  process  of  identifying  candidate  solutions  and  cost 
and  feasibility  analyses  for  each  solution  is  documented  in  the  approved  GCCS 
evolutionary  acquisition  strategy. 

Audit  Response.  The  DISA  comments  are  fully  responsive.  As  stated  by 
DISA  and  the  Deputy  Assistant  Secretary,  the  GCCS  evolutionary  acquisition 
strategy  and  the  Block  IV  Implementation  Plan  will  include  the  documented  cost 
and  feasibility  analyses  of  alternative  solutions. 

c.  Survey  the  users  during  the  application  development  process  to 
ensure  the  final  product  yields  a  user-friendly  interface. 

DISA  Comments.  DISA  concurred  with  the  recommendation,  stating  that  an 
ongoing  installation  of  a  series  of  engineering  drops  has  taken  place  at  various 
locations.  The  installations  allow  users  a  hands-on  evaluation  and  directly 
contribute  to  the  development  effort.  In  addition  to  the  upcoming  functional 
user’s  test  in  June  2002,  users  are  encouraged  to  submit  evaluations  at  any  time 
during  the  development  process.  The  Joint  Staff  is  planning  an  additional  user 
review  panel  designed  to  evaluate  the  requirements  not  incorporated  into  the 
RAS  Output  Tool  schedule.  Additional  requirements  for  future  versions  of  the 
RAS  Output  Tool  will  also  be  solicited. 

Deputy  Assistant  Secretary  Comments.  The  Deputy  Assistant  Secretary  stated 
that  the  recommendation  is  vague  and  does  not  indicate  the  purpose  of  the 
survey.  The  Deputy  Assistant  Secretary  affirmed  his  support  for  user 
involvement  at  appropriate  points  in  the  development  process.  The  ongoing 
process  of  installing  a  series  of  engineering  drops  at  various  locations  allow 
users  an  opportunity  for  hands-on  evaluation  and  directly  contribute  to  the 
development  effort.  In  addition  to  the  upcoming  functional  user’s  test  in  June 
2002,  users  are  encouraged  to  submit  evaluations  at  any  time  during  the 
development  process.  The  Joint  Staff  is  planning  an  additional  user  review 
panel  designed  to  evaluate  the  requirements  not  incorporated  into  the  RAS 
output  tool  schedule.  Additional  requirements  for  future  versions  of  the  RAS 
output  tool  will  also  be  solicited. 

Audit  Response.  The  DISA  comments  are  fully  responsive.  We  modified  the 
recommendation  to  clarify  the  purpose  of  the  survey.  Planned  actions  by  DISA 
and  the  Joint  Staff  will  ensure  user  needs  are  met. 

d.  Support  the  Joint  Staff  in  the  development  and  application  of 
concrete,  measurable,  and  quantifiable  performance  measurements  that  can 
be  accurately  tested  on  the  Readiness  Assessment  System  Output  Tool. 
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DISA  Comments.  DISA  concurred  with  the  recommendation,  stating  that  the 
Assistant  Secretary  tasked  the  Joint  Staff  to  develop  a  GCCS  operational 
requirements  document,  which  will  contain  objective  and  threshold  performance 
parameters  that  are  measurable  and  testable.  DISA  supports  development  of  the 
GCCS  operational  requirements  document  with  participation  in  a 
Joint  Staff-established  and  led  working  group.  With  that  participation,  DISA 
has  already  completed  action  on  this  item. 

Deputy  Assistant  Secretary  Comments.  The  Deputy  Assistant  Secretary  stated 
that  the  Joint  Staff  is  responsible  for  ensuring  that  requirements  are  defined  with 
measurable  and  testable  threshold  and  objective  values.  The  Assistant  Secretary 
tasked  the  Joint  Staff  to  develop  an  approved  GCCS  operational  requirements 
document  with  validated  key  performance  parameters  by  October  2002.  The 
requirements  document  will  include  RAS  Output  Tool  threshold  and  objective 
requirements. 

Audit  Response.  The  DISA  comments  are  fully  responsive.  Actions  taken  by 
DISA,  the  Assistant  Secretary  and  the  Joint  Staff  to  develop  key  performance 
parameters  in  the  GCCS  operational  requirements  document  will  provide  the 
needed  objective  performance  measurements. 

e.  Maintain  comprehensive  records  of  milestones,  goals,  and  key 
decisions  that  document  Readiness  Assessment  System  Output  Tool 
progress. 

DISA  Comments.  DISA  concurred  with  the  recommendation,  stating  that  the 
GCCS  Block  IV  Implementation  Plan  will  document  how  the  RAS  Output  Tool 
will  be  integrated  into  GCCS. 
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Appendix  A.  Scope  and  Methodology 

Scope 


We  reviewed  the  guidance  and  procedures  that  the  Joint  Staff  and  DISA  used  to 
monitor  readiness  issues,  and  the  guidance  and  procedures  that  pertained  to 
requirements  determination  and  fielding  of  appropriate  GCCS  capabilities. 

Work  Performed.  We  reviewed  joint  DoD  guidance  that  governs  application 
and  management  of  GCCS.  We  analyzed  Chairman  of  the  Joint  Chiefs  of  Staff 
(CJCS)  Instruction  3170. OIB,  “Requirements  Generation  System,”  April  2001; 
CJCS  Instruction  672 1.01 A,  “Global  Command  and  Control  Management 
Structure,”  November  2000;  CJCS  Manual  6721.01,  “Global  Command  and 
Control  System  (GCCS)  Functional  Requirements  Evaluation  Procedures,” 
March  1997;  “Readiness  Assessment  System  Business  Plan,”  June  1998; 

“Global  Command  and  Control  System  (GCCS)  Transitional  Evolutionary  Phase 
Implementation  Plan  (EPIP)  for  Phase  III,”  September  2000;  “Global  Command 
and  Control  System  (GCCS)  Transitional  Evolutionary  Phase  Implementation 
Plan  (EPIP)  for  Phase  IV,”  unsigned  draft  dated  June  2001;  “Global  Command 
and  Control  System  (GC(3S)  Evolutionary  Acquisition  Strategy,  Revision  2.2” 
signed  September  18,  2000;  “Guidelines  for  Successful  Acquisition  and 
Management  of  Software-Intensive  Systems,  Volume  1  -  Version  3.0,”  May 
2000;  and  Office  of  the  Secretary  of  Defense  Memorandum  “Requirements  for 
Compliance  with  Reform  Legislation  for  Information  Technology  (IT) 
Acquisitions  (Including  National  Security  Systems),”  May  1,  1997.  The 
documentation  reviewed  covered  the  period  from  March  1997  through  January 
2002. 

High-Risk  Area.  The  General  Accounting  Office  has  identified  several 
high-risk  areas  in  DoD.  This  report  provides  coverage  of  the  DoD  Systems 
Modernization  high-risk  area. 

Methodology 


During  the  audit,  we  evaluated  methods  that  DoD  used  to  develop  and  field 
GCCS  applications.  Specifically,  we  reviewed  policies  and  guidance  that 
govern  Joint  Staff  and  DISA  responsibilities  for  identifying,  documenting,  and 
validating  requirements  for  the  ^^S  Output  Tool.  We  also: 

•  analyzed  minutes  for  Readiness  Assessment  Working  Group  meetings, 
RAS  In-Progress  Reviews,  RAS  Executive  Officer  Working  Group 
meetings,  and  GSORTS  Service  meeting  minutes  from  November  1998 
through  July  2001; 

•  interviewed  officials  from  the  Joint  Staff  and  DISA  responsible  for 
defining,  developing,  and  implementing  the  RAS  Output  Tool; 


21 


•  reviewed  documents  the  readiness  community  within  the  Joint  Staff, 
DISA,  the  Army,  U.S.  Joint  Forces  Command,  Defense  Advanced 
Research  and  Planning  Agency,  and  MITRE,  Incorporated  provided  in 
support  of  the  RAS  Output  Tool  development,  testing,  and  fielding; 

•  reviewed  user  surveys  from  and  solicited  comments  of  potential  users 
throughout  the  U.S.  Central  Command,  the  U.S.  European  Command, 
the  U.S.  Joint  Eorces  Command,  the  U.S.  Pacific  Command,  the  U.S. 
Southern  Command,  the  U.S.  Special  Operations  Command,  the 
Services;  and 

•  analyzed  the  statement  of  work,  periodic  contractor  progress  reports,  and 
funding  documents  that  support  the  $10.5  million  in  RAS  development 
costs  under  contract  number  DCA100-99-C-4019  that  cover 

January  1999  through  January  2002. 

Use  of  Computer-Processed  Data.  We  did  not  use  computer-processed  data  to 
perform  this  audit. 

Audit  Dates  and  Standards.  We  performed  this  audit  from  August  2001 
through  January  2002  in  accordance  with  generally  accepted  government 
auditing  standards.  Accordingly,  we  included  tests  of  management  controls 
considered  necessary. 

Contacts  During  the  Audit.  We  visited  or  contacted  individuals  and 
organizations  within  DoD.  Eurther  details  are  available  on  request. 

Management  Control  Program  Review 


DoD  Directive  5010.38,  “Management  Control  (MC)  Program,”  August  26, 
1996,  and  DoD  Instruction  5010.40,  “Management  Control  (MC)  Program 
Procedures,”  August  28,  1996,  require  DoD  organizations  to  implement  a 
comprehensive  system  of  management  controls  that  provides  reasonable 
assurance  that  programs  are  operating  as  intended  and  to  evaluate  the  adequacy 
of  those  controls. 

Scope  of  the  Review  of  the  Management  Control  Program.  We  reviewed  the 
adequacy  of  management  controls  at  the  Joint  Staff  and  DISA  related  to  GCCS. 
We  reviewed  management  controls  over  GCCS  requirements  generation  and 
program  implementation.  We  also  reviewed  management’s  self-evaluation 
applicable  to  those  controls. 

Adequacy  of  Management  Controls.  We  identified  material  management 
control  weaknesses  within  the  Joint  Staff  and  DISA  as  defined  by  DoD 
Instruction  5010.40.  Specifically,  management  controls  of  the  Joint  Staff  for 
GCCS  oversight  and  requirements  development  did  not  exist. 

Recommendations  2. a.,  2.b.,  and  2.c.,  if  implemented,  will  improve  GCCS 
management  and  development  for  new  applications  in  GCCS.  Also, 
management  controls  in  DISA  for  GCCS  did  not  ensure  that  GCCS  RAS  Output 
Tool  functionality  was  developed,  fielded,  and  documented  with  requirements 
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that  the  users  determined  were  critical.  Recommendation  3. a.,  3.b.,  3.c.,  3.d., 
3.e.,  if  implemented,  will  improve  GCCS  management  and  development  of  new 
applications  in  GCCS.  A  copy  of  the  report  will  be  provided  to  the  senior 
officials  responsible  for  management  controls  at  the  Joint  Staff  and  DISA. 

Adequacy  of  Self-Evaluation.  The  Joint  Staff  did  not  address  GCCS  as  an 
assessable  unit  and,  therefore,  did  not  identify  or  report  the  material 
management  control  weaknesses  identified  by  the  audit.  DISA  addressed 
management  controls  as  an  assessable  unit  and  performed  tests  of  controls  on 
areas  applicable  to  our  audit  objectives.  DISA  officials  did  not  perform 
adequate  tests  of  controls  to  ensure  that  applications  such  as  RAS  had  effective 
program  management  that  includes  appropriately  developed  user  requirements 
and  adequate  baseline  tracking  of  program  status. 

Prior  Coverage 


During  the  last  5  years,  one  General  Accounting  Office  report  that  related  to 
military  readiness  and  two  Inspector  General  of  the  Department  of  Defense 
(IG  DoD)  reports  that  pertained  to  GCCS  were  issued.  The  General  Accounting 
Office  and  IG  DoD  conducted  multiple  reviews  of  GCCS  or  GCCS  applications. 
The  General  Accounting  Office  reports  can  be  accessed  on  the  Internet  at 
http://www.gao.gov.  Unrestricted  IG  DoD  reports  can  be  accessed  at 
http :  // WWW .  dodig .  osd .  mil/ audit/ reports . 

General  Accounting  Office 

General  Accounting  Office  Report  No.  NSI AD-98-68,  “Military  Readiness 
Reports  to  Congress  Provide  Few  Details  on  Deficiencies  and  Solutions,” 

March  1998 

IG  DoD 

IG  DoD  Report  No.  D-2002-084,  “Guidance  for  the  Global  Command  and 
Control  System  Common  Operational  Picture,”  May  1,  2002 

IG  DoD  Report  No.  D-2001-157,  “Global  Command  and  Control  System 
Meteorological  and  Oceanographic  Application,”  July  11,  2001 
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Appendix  B.  RAS  Applications 


The  RAS  Business  Plan  preseribes  an  ineremental  approach  for  replacing 
outmoded  elements  of  GSORTS  with  newer  technology  while  adding  capabilities 
that  the  changing  global  environment  requires.  The  original  RAS  consisted  of 
four  independent  components.  However,  two  of  the  four  components  were 
cancelled.  The  RAS  Output  Tool  is  one  of  the  applications  designed  to  be 
merged  into  the  homogenous  RAS  system.  The  other  two  applications 
originally  planned  for  the  system  were  the  Automated  Joint  Monthly  Readiness 
Review  and  the  Joint  Exercise  Management  Program  III. 

Automated  Joint  Monthly  Readiness  Review.  Through  automated,  near 
real-time  collated,  classified  briefings  and  improved  scenario  analysis,  the 
Automated  Joint  Monthly  Readiness  Review  component  of  RAS  was  intended  to 
streamline  the  Joint  Monthly  Readiness  Review  preparation  process.  As 
originally  envisioned  in  1996,  the  Automated  Joint  Monthly  Readiness  Review 
would  have  served  as  a  combatant  command  Joint  Monthly  Readiness  Review 
reporting  tool.  The  Joint  Monthly  Readiness  Review  reporting  process  matured 
and  integrated  commercial  software,  secure  message,  and  video  teleconference 
technologies,  which  lessened  the  need  for  additional  automation.  On  August  28, 
2000,  the  Joint  Staff  recommended  that  DISA  discontinue  funding  and 
developing  the  Automated  Joint  Monthly  Readiness  Review  and  abandon  plans 
to  make  the  application  a  GCCS  mission  application. 

Joint  Exercise  Management  Program  III.  The  Joint  Exercise  Management 
Program  III  would  have  enabled  the  user  to  import  Joint  Mission  Essential  Task 
List  training  assessment  ratings  that  units  and  combatant  commands  complete 
within  the  Joint  Training  System.  To  help  action  officers  prepare  Joint  Monthly 
Readiness  Report  briefings,  the  Joint  Exercise  Management  Program  III  would 
have  allowed  access  to  Joint  Mission  Essential  Task  List  assessments  from 
within  the  Automated  Joint  Monthly  Readiness  Reporting  system.  Access  to 
Joint  Mission  Essential  Task  List  assessments  was  incorporated  into  the 
Automated  Joint  Monthly  Readiness  Review,  and  the  Joint  Training  Information 
Management  System  replaced  the  Joint  Exercise  Management  Program  III.  The 
Joint  Exercise  Management  Program  III  will  not  be  included  in  RAS. 

RAS  Input  Tool.  The  RAS  Input  Tool  will  allow  near  real-time  reporting, 
improve  the  accuracy  of  GSORTS  data,  and  support  crisis  planning  through 
direct  registration  capabilities  and  stams  reporting  functions.  The  RAS  Input 
Tool  will  enable  Service,  joint,  and  coalition  units  to  submit  user-friendly 
reports  directly  to  the  GSORTS  database.  The  application  is  designed  to  replace 
Service-unique  input  applications  with  a  single  application  that  is  easily  modified 
as  new  requirements  are  added,  thus  eliminating  costly  and  time-consuming 
requirements  that  synchronize  the  system  maintenance  contracts  of  the  Services. 
The  RAS  Input  Tool  allows  efficient  updates  that  help  the  user  keep  pace  with 
rapidly  changing  world  conditions  or  reporting  requirements.  Once  reports  are 
submitted  to  the  master  database,  critical,  tactical,  and  operational  readiness 
information  would  be  immediately  accessible  to  decision  makers  by  way  of  the 
RAS  Output  Tool.  The  Marine  Corps  received  the  RAS  Input  Tool  in 


24 


April  2002  along  with  the  release  of  GCCS  Version  3.4.  As  of  June  19,  2002, 
DISA  projected  that  the  Air  Force  would  receive  the  RAS  Input  Tool  during  the 
third  quarter  of  FY  2003.  The  Army  and  the  Navy  are  not  required  to  adopt  the 
RAS  Input  Tool  because  their  Service-specific  tools  are  designed  to  meet  title  10 
responsibilities  beyond  the  scope  of  the  RAS  Input  Tool. 
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Appendix  C.  Management  Comments  on  the 

Report  and  Audit  Responses 


In  addition  to  the  responses  to  the  recommendations,  the  Deputy  Assistant 
Secretary  of  Defense  (Programs),  Director,  Joint  Staff,  and  the  Director,  DISA 
provided  comments  on  the  audit  report.  Their  comments  are  summarized  and 
presented  by  topic  along  with  the  associated  audit  responses. 

Corrective  Actions 

Deputy  Assistant  Secretary  Comments.  The  Deputy  Assistant  Secretary  stated 
that  DISA  and  the  Joint  Staff  had  already  identified  and  initiated  corrective 
actions  that  mirror  the  recommendations  of  this  audit.  The  Deputy  Assistant 
Secretary  stated  that  plans  for  cost  and  feasibility  analyses  will  be  completed  by 
the  fourth  quarter  of  FY  2002  and  that  the  Joint  Staff  was  tasked  to  develop  an 
operational  requirements  document. 

Joint  Staff.  The  Joint  Staff  stated  that  the  audit  was  limited  to  a  historical 
review  and  failed  to  consider  current  and  future  development  plans  for  the  RAS 
Output  Tool.  The  timeline  and  reengineering  effort  place  the  RAS  Output  Tool 
on  a  development  schedule  that  addresses  previously  discovered  performance 
shortfalls  and  involves  users  from  the  readiness  community. 

DISA  Comments.  DISA  stated  that  the  timeline  for  the  RAS  Output  Tool 
incorporates  a  major  reengineering  effort  that  will  produce  a  robust  and 
responsive  tool  that  will  meet  user  requirements.  DISA  acknowledged  that 
adequate  mechanisms  for  measuring  performance  were  not  yet  established  but  a 
GCCS  operational  requirements  document  would  satisfy  the  requirement. 

Audit  Response.  We  recognize  and  commend  the  efforts  of  the  Assistant 
Secretary  for  reducing  the  earlier  experiments  in  tailoring  the  acquisition 
processes  and  bringing  the  program  into  conformity  with  DoD  acquisition 
management  requirements.  We  also  recognize  the  Joint  Staff  program 
leadership  and  DISA  project  development  efforts.  A  GCCS  operational 
requirements  document  is  clearly  needed.  We  have  some  concerns,  however, 
that  the  operational  requirements  document  and  the  performance  standards  will 
be  developed  to  justify  already  completed  work  rather  than  to  satisfy  current 
needs.  We  will  address  our  concerns  in  the  audit  followup  process. 
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Readiness  Assessment  System 

Joint  Staff  Comments.  The  Joint  Staff  stated  that  the  draft  report  failed  to 
recognize  the  distinction  between  the  RAS  and  the  RAS  Output  Tool. 

The  Joint  Staff  also  stated  that  the  historical  view  presented  in  the  report  failed 
to  consider  current  and  future  development  plans  for  the  RAS  Output  Tool. 

DISA  Comments.  DISA  stated  that  the  report  unfairly  applied  standards 
intended  for  the  RAS  to  the  RAS  Output  Tool. 

Audit  Response.  The  report  establishes  the  relationship  between  the  RAS 
Output  Tool  and  the  RAS  discussed  on  page  2  of  this  report.  The  additional 
RAS  applications  are  discussed  in  Appendix  B.  The  Joint  Staff  and  DISA  could 
not  provide  standard  criteria  governing  the  development  of  the  RAS  Output  Tool 
that  was  more  current  than  the  June  1998  RAS  Business  Plan.  The  Joint  Staff 
established  the  requirement  in  a  June  1998  message  that  the  RAS  Output  Tool 
replace  Service  and  combatant  command  tools  while  retaining  JRAMS 
functionality.  At  no  point  in  the  report  do  we  hold  the  RAS  Output  Tool  to 
standards  intended  for  other  applications.  Unless  otherwise  qualified,  we 
interpreted  the  term  “RAS”  in  the  RAS  Business  Plan  to  mean  all  components 
of  the  RAS. 

User  Requirements 

Joint  Staff  Comments.  The  Joint  Staff  stated  that  the  lack  of  community 
awareness  noted  in  the  survey  is  not  surprising  considering  personnel  turnover 
rates  compared  to  RAS  development  timeline.  Users  do  not  normally 
participate  in  the  developmental  phase  of  the  system  acquisition  process,  nor  do 
they  conduct  formal  developmental  testing  because  they  would  likely  provide 
conflicting  or  confusing  feedback  that  would  be  more  likely  to  disrupt,  not 
enhance  the  process.  RAS  requirements  were  briefed  as  necessary  to  the  Global 
Command  and  Control  Readiness  Working  Group  and  the  Global  Command  and 
Control  Readiness  Working  Group  User  Review  Panel.  Since  functional 
versions  of  RAS  Output  Tool  have  been  provided  to  various  commands,  perhaps 
the  Inspector  General,  Department  of  Defense  failed  to  interview  the  right 
people.  Although  the  Joint  Staff  acknowledged  that  JRAMS  had  superior  speed 
to  RAS,  the  Joint  Staff  also  criticized  the  comparison  to  the  RAS  Output  Tool. 
The  Joint  Staff  disagreed  that  DISA  and  the  Joint  Staff  have  not  provided 
sufficient  assurance  that  the  RAS  Output  Tool  will  satisfy  user  requirements. 

The  Joint  Staff  amassed  a  total  of  734  requirements  for  ^S  Output  Tool  from 
combatant  commanders  and  Services.  The  list  is  constantly  validated,  updated 
and  documented,  with  395  requirements  incorporated  in  RAS  Output  Tool  4.0. 

DISA  Comments.  DISA  agreed  that  potential  users  lacked  the  knowledge  and 
involvement  with  the  RAS  Output  Tool  at  the  time  of  the  audit.  However, 

DISA  is  currently  developing  the  RAS  Output  Tool  based  on  a  total  of  734 
requirements  derived  from  Combatant  Commands  and  Services.  These 
requirements  originated  from  a  series  of  Joint  Application  Development  sessions 
held  between  mid- 1997  and  1999  with  Combatant  Command  and  Service 
representatives  and  approved  by  the  Joint  Staff.  DISA  acknowledged  that  the 
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lack  of  speed  was  a  major  driving  force  behind  the  recommendation  to 
reengineer  the  RAS  Output  Tool  in  July  2001.  DISA  added  that  the  RAS 
Output  Tool  is  slower  than  JRAMS  because  the  RAS  Output  Tool  must  search 
and  provide  more  data.  DISA  recently  provided  pre-release  engineering  drops 
of  the  RAS  Output  Tool  to  several  users. 

Audit  Response.  We  contacted  users  identified  by  the  Joint  Staff  as 
representing  the  readiness  user  community.  Neither  the  Joint  Staff  nor  DISA 
could  document  that  the  RAS  Output  Tool  requirements  were  generated  from 
the  Joint  Application  Development  sessions  or  that  RAS  users  other  than  the 
Joint  Staff  constantly  validated  and  updated  requirements.  The  Joint  Staff 
re  validation  in  May  2001  of  user  requirements  did  not  include  end-user 
involvement.  The  Joint  Staff  stated,  in  response  to  renumbered 
Recommendation  2. a.,  that  as  a  developmental  necessity  the  RAS  Output  Tool 
requirements  were  “locked-in  and  will  not  change  because  software  engineers 
cannot  hit  a  moving  target  resulting  from  changing  requirements.  ”  Such  a 
policy  appears  reasonable  in  cases  where  fielding  dates  have  not  slipped  more 
than  2  years  and  development  initiated  based  on  an  approved  operational 
requirements  document.  In  addition,  users  were  not  able  to  develop  an 
expectation  for  RAS  Output  Tool  system  speed,  due  to  the  limited  user 
involvement,  and  only  had  JRAMS  performance  to  draw  a  comparison. 

Air  Force  guidance  on  software  process  improvement.  Guidelines  for  Successful 
Acquisition  and  Management  of  Software-Intensive  Systems,  Volume  1  - 
Version  3.0,  May  2000,  emphasizes  the  need  for  the  user  and  developer  to  work 
together: 


As  long  as  programmers  are  writing  code,  they  are  making  design  decisions, 
just  at  a  more  detailed  level.  Many  of  these  details  will  impact  the  usability 
and  performance  of  the  system,  just  not  at  a  high  enough  level  for  the  people 
who  wrote  the  requirements  to  be  aware  of  them.  The  field  users  of  sueh 
systems,  however,  will  almost  always  find  that  systems  developed  blindly  from 
requirements  documents  are  inconvenient  and  unwieldy  in  operational  use. 

Truly  superior  usability  can  only  be  obtained  when  the  developers  have  an  in- 
depth  knowledge  of  actual  field  conditions.  While  suppliers  should  start  from 
official  requirements,  these  must  be  recognized  as  a  starting  point  and  that 
much  more  detailed  knowledge  is  required  before  the  system  can  actually  be 
built.  The  key  is  to  make  the  supplier  responsible  for  devising,  defining,  and 
using  a  process  that  uneovers  true  operational  requirements. 

RAS  Business  Plan 

Joint  Staff  Comments.  Although  the  Joint  Staff  acknowledged  issues  involving 
validation  of  requirements,  the  Joint  Staff  nonconcurred  with  conclusions  drawn 
from  the  RAS  Business  Plan  and  the  inelusion  of  the  Seeretary  of  Defense 
comments  which  applied  to  future  RAS  development.  The  Joint  Staff  also  stated 
that  the  Army  eomments  applied  to  the  full  RAS  business  plan;  not  speeifically 
the  RAS  Output  Tool.  The  Army  was  worried  that  RAS  Business  Plan  was  too 
invasive,  not  that  RAS  Output  Tool  would  not  meet  their  need.  The  Joint  Staff 
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stated  that  Service  comments  were  incorporated  into  the  RAS  Business  Plan  and 
agreed  with  DISA  that  standards  for  the  entire  RAS  were  applied  to  the  RAS 
Output  Tool. 

DISA  Comments.  DISA  stated  that  the  draft  report  misrepresented  comments 
that  the  Secretary  of  Defense  and  the  Army  provided.  The  Secretary  of  Defense 
comments  were  to  be  included  in  a  section  of  the  RAS  Business  Plan  that 
pertains  to  future  applications.  DISA  contends  the  Army  had  no  concerns  about 
the  ability  to  of  the  RAS  Output  Tool  meeting  Army  needs. 

Audit  Response.  Although  the  Secretary  of  Defense  comments  were  included 
in  a  section  of  the  RAS  Business  Plan  that  pertains  to  future  applications,  those 
comments  addressed  the  RAS  without  qualification.  Unless  stated  otherwise, 
we  interpret  references  to  the  '‘RAS”  to  include  all  RAS  applications.  We  also 
believe  the  Army  comments  that  DISA  provided  in  support  of  the  DISA  position 
speak  for  themselves. 

The  recurring  concepts  supporting  RAS  are  “commanders  at  all  levels”  and  “near 
real  time”  readiness  information.  Army  believes  a  more  thorough  mission  needs 
analysis  is  needed  to  determine  more  precisely  the  information  timeliness  needs  at 
each  level  of  command,  and  to  target  that  information  to  the  needs  of  the  recipient. 

For  instance,  policy  makers  do  not  need  minute  details  or  24-hour  readiness 
updates  to  carry  out  policy-making  responsibilities.  Additionally,  in  the  world  of 
deliberate  planning,  current  readiness  data  is  not  useful  .  .  .  Only  in  the  sourcing  of 
contingency  operation  force  requirements  is  current  readiness  a  factor. 

RAS  seems  to  be  driven  more  by  the  fact  that  the  technology  to  achieve  a 
directed  telescope  to  the  forward  foxhole  will  soon  be  available  than  by  any 
validated  need.  Army  acknowledges  that  RAS  consists  of  a  grouping  of  systems 
and  software,  which  are  already  in  some  state  of  development  or  fielding,  many  of 
which  are  already  funded  and  are  intended  to  be  resident  in  the  GCCS/GCSS 
suites.  However,  Army  believes  that  the  true  cost  of  a  comprehensive  RAS  has  not 
been  determined.  Finally,  Army  is  concerned  that  the  potential  for 
micromanagement  inherent  in  RAS  represents  an  intentional  or  unintentional 
intrusion  into  the  Services’  title  X  responsibilities  to  provide  ready  forces  to  the 
combatant  commanders. 


Testing 

Joint  Staff  Comments.  The  Joint  Staff  stated  that  the  RAS  Output  Tool  was 
tested  on  numerous  occasions  and  that  scheduling  delays  were  a  direct  result  of 
system  failures  identified  during  testing.  The  Joint  Staff  stated  that  U.S.  Joint 
Forces  Command  comments  on  the  adequacy  of  the  functional  qualification  tests 
were  typical  of  users  who  apply  standards  for  mature  systems  to  evolving 
systems.  Involving  users  too  early  in  the  process  usually  erodes  their 
confidence  in  the  system  and  can  have  the  opposite  effect  as  recommended  by 
the  Office  of  the  Inspector  General,  Department  of  Defense. 

DISA  Comments.  DISA  stated  that  when  users  are  involved  too  early  in  the 
testing  process,  confidence  in  the  final  system  could  be  irretrievably  damaged. 
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Users  not  eminently  familiar  with  systems  development  methodology  would 
provide  confusing  or  conflicting  feedback.  However,  once  the  product  reaches 
the  latter  stages  of  developmental  testing  and  clearly  during  the  operational 
testing  phase,  user  input  is  essential.  The  Joint  Staff  mandate  ensures  that 
functional  user  and  operational  testing  heavily  rely  on  user  representatives.  For 
example,  to  obtain  feedback  on  the  JRAMS  fonctions  being  incorporated  into 
the  RAS  Output  Tool,  DISA  asked  the  U.S.  Joint  Forces  Command  to  view  a 
RAS  Output  Tool  demonstration.  The  U.S.  Joint  Forces  Command 
demonstration  resulted  in  eight  new  validated  requirements.  A  functional  user 
test  was  scheduled  for  June  2002. 

Audit  Response.  Developing  a  product  without  continued  user  involvement 
increases  the  risk  that  the  product  would  not  satisfy  user’s  needs,  particularly 
after  a  long  development  period.  While  there  are  times  when  lessened  user 
involvement  is  understandable,  a  multiyear  cessation  is  harder  to  defend  without 
verifying  a  continued  user’s  need.  Neither  the  Joint  Staff  nor  DISA  could 
provide  any  documentation  that  users  other  than  the  Joint  Staff  and  U.S.  Joint 
Forces  Command  had  consulted  in  the  development  of  the  RAS  Output  Tool. 
Only  the  Joint  Staff  appears  to  have  been  involved  in  the  May  2001 
requirements  validations.  We  support  the  recent  activity  intended  to  expose  the 
users  to  the  pre-release  engineering  drop  of  the  proposed  RAS  Output  Tool  and 
the  planned  functional  user  testing. 

Documentation  of  Decisions 

Joint  Staff  Comments.  The  Joint  Staff  stated  that  a  review  of  Joint  Staff 
records  did  not  reveal  the  documentation  that  would  support  a  search  for 
candidate  solutions  and  analysis  of  alternatives  took  place.  By  the  time  the 
Secretary  of  Defense  approved  the  RAS  Business  Plan,  the  milestone  decision 
was  made  that  DISA  proceed  with  development  of  the  RAS  Output  Tool  by 
building  on  the  work  the  original  development  contractor  began.  There  is  also  a 
volume  of  correspondence  covering  the  period  from  January  1998  to  July  2000, 
which  provides  considerable  detail  into  the  origin,  shortcomings  and  missed 
milestones  for  the  RAS  Output  Tool.  There  is  a  much  smaller  volume  covering 
the  current  contract  covering  RAS  applications  in  general,  including  the  RAS 
Output  Tool. 

DISA  Comments.  DISA  stated  that  critical  development  and  fielding  decisions 
have  been  documented  with  a  great  deal  of  correspondence  that  has  provided 
considerable  detail  into  milestones  and  decisions. 

Audit  Response.  Neither  the  Joint  Staff  nor  DISA  provided  any  documentation 
that  supported  any  analysis  of  alternatives,  user  involvement  in  revalidation,  or 
other  documentation  from  application  development  sessions.  DISA  and  the 
Joint  Staff  did  not  demonstrate  in  the  documentation  provided  that  an  analysis 
had  taken  place  of  cost,  performance,  and  schedule  that  applied  to  the  RAS 
Output  Tool.  The  Joint  Staff  provided  evidence  of  a  DISA  request  for 
requirement  revalidation.  However,  the  Joint  Staff  response  did  not  show  user 
involvement  in  the  re  validation. 


30 


Development  of  Alternative  Applications 

Joint  Staff  Comments.  The  Joint  Staff  disagreed  that  delays  in  fielding  of  the 
RAS  Output  Tool  required  the  combatant  commands  and  Services  to  either 
develop  new  applications  or  modify  existing  applications.  The  Joint  Staff  stated 
that  the  RAS  Output  Tool  was  never  intended  to  replace  combatant  command  or 
Service  tools,  and  that  the  RAS  Input  Tool  was  the  only  one  of  the  four  original 
RAS  applications  designed  to  do  so. 

DISA  Comments.  DISA  disagreed  with  the  connection  between  RAS  Output 
Tool  delays  and  the  development  of  combatant  command  or  Service  tools.  The 
RAS  Input  Tool  was  designed  to  replace  selected  Service  input  tools,  but  the 
RAS  Input  Tool  is  not  an  application  for  performing  readiness  analysis. 

Audit  Response.  The  Joint  Staff  message  that  terminated  JRAMS  development 
in  1998  states  that  the  RAS  Output  Tool  will  contain  all  of  the  JRAMS 
functions.  The  U.S.  Joint  Forces  Command  has  continued  to  fund  JRAMS 
development  because  they  believe  that  the  RAS  Output  Tool  will  not  fulfill  the 
needs  that  JRAMS  supplies.  The  RAS  Business  Plan  states  that  the  RAS  Output 
Tool  will  include  the  functions  of  the  Army  Readiness  Management  System, 
which  the  Army  continues  to  run  and  maintain.  The  Air  Force  is  developing  the 
Predictive  Readiness  Assessment  System.  Predictive  functionality  of  the  RAS 
Output  Tool  will  provide  unit  trends  over  a  period  of  time  and  will  project 
readiness  levels  of  particular  units. 

Performance  Measures 

Joint  Staff  Comments.  DISA  and  the  Joint  Staff  did  establish  effective 
performance  measures  to  monitor  development  and  measure  success,  or  failure, 
against  a  specific  set  of  criteria.  Every  single  approved  requirement  stated  for 
RAS  Output  Tool  version  4.0  has  a  documented  test  procedure  and  an  expected 
outcome. 

DISA  Comments.  DISA  agreed  that  an  adequate  mechanism  for  measuring 
performance  was  not  established.  In  November  2000,  a  GCCS  program 
management  review  recognized  the  need  for  a  GCCS  operational  requirements 
document  which  will  identify  objectives  and  thresholds  for  operational 
requirements,  as  well  as  key  performance  parameters  for  the  GCCS  system,  to 
include  RAS  Output  Tool.  The  Joint  Staff  is  currently  developing  a  GCCS 
operational  requirements  document. 
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Appendix  D.  Report  Distribution 


Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  (Comptroller)/Chief  Financial  Officer 
Deputy  Chief  Financial  Officer 
Deputy  Comptroller  (Program/Budget) 

Under  Secretary  of  Defense  for  Personnel  and  Readiness 

Assistant  Secretary  of  Defense  (Command,  Control,  Communications,  and  Intelligence) 


Joint  Staff 

Director,  Joint  Staff 

Director,  Operations  (J-3) 

Director,  Command,  Control,  Communications,  and  Computers  (J-6) 


Department  of  the  Army 

Auditor  General,  Department  of  the  Army 

Department  of  the  Navy 

Naval  Inspector  General 

Auditor  General,  Department  of  the  Navy 


Department  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  (Financial  Management  and  Comptroller) 
Auditor  General,  Department  of  the  Air  Force 


UniUed  Commands 

Commander,  U.S.  European  Command 
Commander,  U.S.  Pacific  Command 
Commander,  U.S.  Joint  Forces  Command 
Commander,  U.S.  Southern  Command 
Commander,  U.S.  Central  Command 
Commander,  U.S.  Space  Command 
Commander,  U.S.  Special  Operations  Command 
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Unified  Commands  (cont’d) 

Commander,  U.S.  Transportation  Command 
Commander,  U.S.  Strategic  Command 


Other  Defense  Organization 

Director,  Defense  Information  Systems  Agency 


Non-Defense  Federal  Organization 

Office  of  Management  and  Budget 


Congressional  Committees  and  Subcommittees,  Chairman  and 
Ranking  Minority  Member 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 
Senate  Committee  on  Armed  Services 
Senate  Committee  on  Governmental  Affairs 
House  Committee  on  Appropriations 

House  Subcommittee  on  Defense,  Committee  on  Appropriations 
House  Committee  on  Armed  Services 
House  Committee  on  Government  Reform 

House  Subcommittee  on  Government  Efficiency,  Financial  Management,  and 
Intergovernmental  Relations,  Committee  on  Government  Reform 
House  Subcommittee  on  National  Security,  Veterans  Affairs,  and  International 
Relations,  Committee  on  Government  Reform 
House  Subcommittee  on  Technology  and  Procurement  Policy,  Committee  on 
Government  Reform 
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Assistant  Secretary  of  Defense  (Command, 
Control,  Communications,  and  Intelligence) 


OFFICE  OF  THE  ASSISTANT  SECRETARY  OF  DEFENSE 
6000  DEFENSE  PENTAGON 
WASHINGTON,  DC  20301-6000 

May  28.  2001 


MEMORANDUM  FOR  INSPECTOR  GENERAL,  DEPARTMENT  OF  DEFENSE 

SUBJECT:  Audit  Report  on  Global  Command  and  Control  System  -  Readiness 
Assessment  System  Output  Tool  (Project  No.  D-2001 LG-01 01.01) 


Your  memOTandum,  March  2002.  requested  review  and  comment  for  the 
subject  proposed  audit  report.  OASD(C3I)  comments  are  attached. 

This  secoiid  in  a  series  of  DOD IG  reports  on  Global  Command  and  Control 
System  (GCCS)  unfortunately  selected  a  topic  that  DISA  and  the  Joint  Staff  had  already 
identified  and  initiated  corrective  actions.  The  actions  initiated  by  DISA  and  the  Joint 
Staff  match  the  recommendations  in  this  report,  and  have  largely  been  completed-  It  is 
not  appropriate  at  this  point  to  conduct  an  additional  unplanned  validation  of  user 
requirements. 

OASD(C3I).  DISA  and  the  Joint  Staff  have  identified  similar  problems  with  the 
GCCS  Joint  Operational  Planning  and  Execution  System  (JOPES).  Actions  have  been 
initiated  to  correct  these  problems  also. 

The  OASD(C3I)  point  of  contact  for  this  matter  is  COL  Michael  Krieger,  703-607- 
0259,  who  is  assigned  to  the  C2  Programs  Directorate . 


Dr.  Michael  S.  Frankel 

Deputy  Assistant  Secretary  of  Defense 

(Programs) 


Attachment 
As  Stated 


a 
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Final  Report 
Reference 


Revised  and 
renumbered 
as 

Recommen¬ 
dation  2.  a. 
Page  15 


Renumbered 

as 

Recommen¬ 
dation  2.b. 
Page  16 


OASD  (C3I)  Comments  on  the  E)raft  Audit  Report  on  Global  Command  and  Control 
System  -  Readiness  Assessment  System  Output  Tool  (Project  No.  D-2001LG-0101.01) 


Direct  quotes  from  the  DODIG  report  are  presented  in  Italics  with  page  numbers 
from  the  draft  audit. 

Recommendations : 

1.  Recommendations  for  the  Director  for  Operations,  Joint  Staff; 

a.  ""Revalidate  the  original  Global  Command  and  Control  System 
Readiness  Assessments  Output  Tool  requirements  with  current  users  within 
the  Unified  Commands,  Services,  and  Defense  Agencies  and  ensure  that 
future  GCCS  user  requirements  are  validated  and  implemented  in 
accordance  with  current  guidance. "  (Page  14) 

C3I  non-concurs,  JS  J3  DDGO-R  already  revalidated  the  full  set  of  RAS  OT  user 
requirements  in  May  2001  in  preparation  for  the  Functional  Qualification  Testing  in  June 
2001.  The  results  were  briefed  to  the  Readiness  Assessment  System  Executive  Steering 
Committee  (RAS  ESC)  in  July  2001 .  The  Joint  Staff  and  DISA  will  review  the  full  user 
requirement  list  again  in  May  2002  in  preparation  for  RAS  OT  Functional  User  Testing 
scheduled  for  June  2002.  Joint  Staff  has  indicated  they  will  review  user  requirements 
slated  for  RAS  OT  versions  beyond  4.0  as  well  as  any  additional  requirements  collected. 
Joint  Staff  has  indicated  this  effort  will  include  all  Combatant  Command,  Combat 
Support  Agency,  and  Service  readiness  divisions. 

The  recommendation  regarding  future  requirements  is  confusing.  If  the  reference 
is  to  RAS  OT  requirements  only,  delete  the  phrase  “GCCS  user  requirements’"  and 
replace  with  “RAS  OT  user  requirements.  If  the  reference  is  to  user  requirements  for  all 
of  GCCS,  C3I  non-concurs.  As  an  ACAT-IAM  program,  GCCS  user  requirements  are 
documented  in  a  JROC  approved  ORD,  The  Joint  Staff  J-3  is  transitioning  from  a  GCCS 
Requirements  Identification  Document  (RID)  to  a  JROC  approved  GCCS- Joint  ORD. 
The  GCCS-Joint  ORD  is  scheduled  to  go  out  for  stage  I  JROC  coordination  in  June  2002. 
A  JROC  approved  ORD  will  ensure  GCCS-Joint  user  requirements  are  validated  and 
implemented  in  accordance  with  current  guidance. 

b.  ""For future  GCCS  Readiness  applications,  search  Government  and 
commercial  sources  for  applications  that  will  satisfy  requirements  and 
document  the  results  of  those  searches.  ”  (Page  14) 

C3I  non-concurs.  Additional  efforts  in  this  direction  would  be  redundant.  Current 
DUSD  Personnel  and  Readiness  initiatives  to  conceptualize  and  develop  an  improved 
readiness  reporting  system  for  DOD  are  fulfilling  this  requirement.  The  Joint  Staff  and 
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Final  Report 
Reference 


DISA  are  involved  in  this  effort.  The  Joint  Staff  is  also  involved  in  a  similar  process  as 
part  of  the  GCCS  ORD  development. 

C-  '"Develop  performance  standards  that  as  a  minimum,  provide  a  way  of 
assessing  the  performance,  cost,  schedule,  technical  quality  and  reliability, 
and  user  satisfaction  and  formally  report  on  the  progress  of  the  Readiness 
Assessment  System  Output  Tool  in  meeting  those  standards”.  (Page  15) 

C3I  non-concurs.  The  recommendation  tasks  the  Joint  Staff  to  perform  activities 
outside  their  established  role.  The  Joint  Staff  is  responsible  to  identify,  validate,  and 
prioritize  GCCS  RAS  OT  requirements,  as  well  as  provide  measurable  and  testable 
threshold  and  objective  performance  parameters.  It  is  the  GCCS  PM's  responsibility  to 
manage  cost,  schedule  and  performance,  and  report  in  accordance  with  DoD  5000,2-R. 
PA&E  independently  assesses  the  program  costs  through  review  of  the  program’s 
economic  analysis.  The  GCCS  program  manager  (PM)  is  responsible  for  building  the 
plan  to  develop,  integrate  test,  accredit  and  field  the  Readiness  Assessment  System 
Output  Tool  (RAS  OT)  as  part  of  GCCS.  OSD  provides  appropriate  acquisition 
oversight  of  the  program  while  the  Joint  Staff  and  users  provide  an  assessment  on  the 
functionality  delivered  in  RAS  OT. 

2.  Recommendations  for  the  Director,  Defense  Information  Systems  Agency, 

a.  "Pause  development  of  the  Readiness  Assessment  System  Output  Tool 
until  the  Joint  Staff  fully  addresses  requirements  development,  user 
satisfaction,  program  documentation,  and  performance  measurement.  ” 
(Page  15) 

C3I  non-concurs.  A  delay  for  the  sake  of  staff  analysis  would  be  inappropriate, 
costly,  and  counterproductive  to  delivering  capability  to  the  user  community.  The  Joint 
Staff  and  DISA  have  taken  appropriate  actions  to  ensure  delivery  of  the  required  RAS  OT 
capabilities.  A  pause  was  taken  during  the  spring  and  early  summer  of  2001  under  the 
auspices  of  the  RAS  ESC.  Based  upon  a  full  review  and  validation  of  requirements,  a 
thorough  Functional  Qualification  Test,  and  feedback  from  the  user  community,  the  RAS 
ESC  directed  DISA  in  July  2001  to  reengineer  the  RAS  OT.  Beginning  with  the 
installation  in  October  2001,  pre-release,  functioning  versions  of  the  RAS  OT  are  resident 
at  JS  J3  DDGO-R,  JFCOM,  EUCOM,  TRANSCOM,  HQ  US  Navy,  HQ  US  Air  Force, 
HQ  US  Army,  HQ  USMC,  MARFORLANT  and  MARFORRES.  The  first  test  of  an 
operationally  representative  system  involving  users  from  Combatant  Commands  and 
Services  is  scheduled  for  June  2002.  At  this  point  another  development  pause  would 
further  exacerbate  the  same  condition  that  has  brought  the  most  negative  scrutiny,  more 
scheduling  delays.  Previous  delays  were  caused  by  failures  of  the  software  to  meet  user 
requirements. 


Renumbered 

as 

Recommen¬ 
dation  2.C. 
Page  16 


Revised  and 
renumbered 
as 

Recommen¬ 
dation  3.  a. 
Page  17 


37 


Final  Report 
Reference 
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as 

Recommen¬ 
dation  3.b. 
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Revised  and 
renumbered 
as 

Recommend 
ation  3.C. 
Page  19 


Renumbered 

as 
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dation  3.d. 
Page  19 


Deleted 


b.  ^'Develop  and  document  cost  and  feasibility  analyses  of  solutions  that 
are  identified  by  the  Joint  Staff. "  (Page  15) 

C3I  non-concurs.  The  Joint  Staff  is  not  responsible  for  identifying  solutions.  The 
Joint  Staff  is  responsible  for  developing  GCCS  requirements.  The  process  to  identify 
candidate  solutions  and  develop  cost  and  feasibility  of  each  solution  is  documented  in  the 
approved  GCCS  Evolutionary  Acquisition  Strategy.  The  GCCS  Block  IV 
Implementation  Plan  (formally  known  as  the  Phase  IV  Evolutionary  Phase 
Implementation  Plan)  will  include  the  capabilities  that  successfully  pass  this  analysis  and 
show  how  they  are  to  be  integrated  into  GCCS.  This  plan  is  anticipated  to  be  completed 
and  provided  to  the  GCCS  Integrating  Integrated  Product  Team  (IIPT)  in  the  summer  of 
2002  and  submitted  for  GCCS  Milestone  Decision  Authority  (MDA)  approval  in 
Quarter  FY  02. 


c.  ^'Survey  the  users  during  the  application  development  process.  ”  (Page 
15) 

C3I  non-concurs.  The  reconunendation  is  vague.  It  does  not  indicate  the  purpose 
of  the  survey.  C3I  supports  user  involvement  at  appropriate  points  in  the  development 
process.  This  process  is  ongoing  with  the  current  installation  of  a  series  of  engineering 
drops  at  various  locations  (see  response  in  paragraph  2.a.  above).  These  installations 
allow  users  a  hands-on  evaluation  thus  contributing  directly  to  the  development  effort.  In 
addition  to  the  upcoming  functional  user’s  test  in  June  2002,  users  are  encouraged  to 
submit  evaluations  at  any  time.  Also  the  Joint  Staff  is  planning  an  additional  User 
Review  Panel  to  evaluate  the  requirements  that  have  not  been  incorporated  into  the  RAS 
OT  schedule.  Additional  requirements  for  future  versions  of  the  RAS  OT  will  also  be 
solicited. 


d.  "^Support  the  Joint  Staff  in  the  development  and  application  of  concrete, 
measurable,  and  qualitative  performance  measurements  that  can  be 
accurately  tested  on  the  Readiness  Assessment  System  Output  Tool.  ”  (Page 
15) 

C3I  concurs  with  comment.  As  the  requirements  authority  the  Joint  Staff  is 
responsible  for  ensuring  that  requirements  are  defined  with  measurable  and  testable 
Threshold  and  Objective  values.  OASD-C3I  tasked  the  Joint  Staff  to  develop  a  JROC- 
approved  GCCS  Operational  Requirements  Document  (ORD)  with  validated  Key 
Performance  Parameters  (KPPs)  by  Oct  02.  This  ORD  will  include  RAS  (OT)  Threshold 
and  Objective  requirements. 

e.  '"Support  the  Joint  Staff  in  developing  the  RAS  (OT)  for  performance 
standards  that,  at  a  minimum,  provide  a  way  of  assessing  performance,  cost 
schedule,  technical  quality  and  reliability,  and  user  satisfaction  and 
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formally  report  on  how  the  Readiness  Assessment  System  Output  Tool  is 
meeting  those  standards,  ”  (Page  15) 

C3I  non-concurs.  The  recommendation  fails  to  recognize  the  distinction  of  the 
separate,  yet  complementary,  roles  of  the  Joint  Staff,  as  Requirements  Authority,  and 
DISA,  as  GCCS  ftogram  Manager  nor  the  formal  processes  already  in  place  for  program 
planning  and  reporting.  The  Joint  Staff  is  responsible  for  developing  GCCS 
requirements.  It  is  the  GCCS  PM's  responsibility  to  manage  cost,  schedule  and 
performance,  and  report  in  accordance  with  DoD  5000,2-R,  PA&E  independently 
assesses  the  program  costs  through  review  of  the  program’s  economic  analysis.  The 
operational  assessment  (OA)  or  operational  test  (OT)  will  report  on  how  the  Readiness 
Assessment  System  Output  Tool  is  meeting  those  standards. 
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Joint  Staff  Comments 


THE  JOINT  STAFF 
WASHimmuiDc 


Reply  ZIP  Code:  IXJSM-0489-02 

203 18-0300  23  May  2002 


MEMORANDUM  FOR  THE  INSPECTOR  GENERAL,  DEPARTMENT  OP 
DEFENSE 

Subject:  Proposed  DOD  Inspector  General  Report  on  Global  Command  and 

Control  System  Readiness  Assessment  System  Output  Tool  (RAS-OT) 
(D2001LG-010L01) 

1.  Thank  you  for  the  opportunity  to  review  the  draft  proposed  DOD  IG  reports 
While  portions  of  the  report  have  merit,  I  cannot  concur  in  many  of  its  findings 
and  recommendations. 

2.  The  emphasis  of  the  report  is  on  management  policy.  I  recognize  the  need 
to  correct  management  deHciencies  and  the  Joint  Staff  is  working  with  DISA 
and  the  RAS-OT  contractor  to  resolve  them.  However,  I  disagree  with  those 
recommendations  that— if  foUowed—would  result  in  further  delays  to  the  RAS- 
OT  development  process.  In  addition,  I  believe  that  the  report  does  not 
adequately  recognize  progress  made  in  improving  the  RAS-OT  program  over  the 
last  12  months.  The  enclosure  provides  detedled  comments. 

3.  The  Joint  Staff  point  of  contact  is  LTC  Mike  Baker,  J3,  693-5475/8191, 

JOHN  P.  ABIZAID 
Lieutenant  General,  USA 
Director,  Joint  Staff 

Enclosure 

Reference: 

1  DODIG  memorandum,  20  March  2002,  "Audit  Report  on  the  Global 
Command  System  Readiness  Assessment  System  Output  Tool  (Project 
No.  D2001LG-0101.01) 
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ENCLOSURE 

latroduction 

This  document  is  the  official  Joint  Staff  response  to  DOD  IG  Project 
D2001 LG-01 01.01;  Global  Command  and  Control  System  Readiness 
Assessment  System  Output  Tool.  Direct  quotes  from  the  DOD  IG  report  are 
presented  in  italics  with  page  numbers  from  the  draft  audit.  Other  quotes  are 
listed  in  italics  as  well  with  appropriate  citations. 

(Page  1,  paragraph  4)  VMnt  Stttff,  The  Joint  Staff,  as  the  project  sponsor 
for  all  GCCS  applications,  represents  the  needs  of  the  users.  Throughout  the 
requirements  validation  and  approval  process,  the  Joint  Staff  ident^s  and 
prioritizes  Junctional  requirements  that  must  be  satisfied  by  GCCSf 

Objectives 

(Page  3,  paragraph  4,  4th  line)  ' .  .  .  The  specific  objective  for  this  segment 
of  the  audit  ufos  to  assess  management  and  oversight  of  the  Readiness 
Assessment  System  Output  Tool^ 

The  Joint  Staff  understands  the  objective  of  this  audit.  It  is  important  to 
ensure  the  findings  and  recommendations  in  this  report  apply  to  specific 
requirements  and  expectations  applicable  to  the  Readiness  Assessment  System 
Output  Tool  (RAS-OT)  as  stated  in  the  DOD  IG  objective. 

Important  historical  facts:  These  facts  will  be  important  when  addressing 
DOD  IG  findings  foimd  later  in  the  discussion.  According  to  the  1998  Business 
Plan,  approved  by  the  Secretaiy  of  Defense  in  1999,  RAS  would  have  combined 
four  independent  projects  in  Phase  II  (the  phase  under  review).  Those  projects 
were  1)  Automated  Joint  Monthly  Readiness  Review  (AJMRR);  2)  Global  Status 
of  Resoiorces  and  Training  System  Enhanced  Output  (GSORTS  (E)(0)]  now 
commonly  referred  to  as  RAS-OT;  3)  Joint  Ebcercise  Management  Program  III 

Enclosure 
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(JEMP  III)  and  4)  Global  Status  of  Resources  and  Trauning  System  Enhanced 
Input  (GSORTS  (E)(1)).  This  is  acknowledged  on  page  2  of  the  DOD  IG  draft 
report. 

Point  One:  The  AJMRR  was  canceled.  The  JEMP  III  was  removed  from  the 
Readiness  Assessment  System  and  placed  imder  a  training  management 
program  within  J7.  GSORTS  (E)(0),  now  called  RAS-OT,  was  never  designed  to 
carry  the  full  load  of  the  original  RAS  concept  as  outlined  in  the  Business  Plan. 
The  DOD  IG  should  not  apply  standards  set  for  the  holistic  RAS  concept  to 
RAS-OT  as  a  stand-alone  application. 

Point  Two:  There  is  nothing  in  the  RAS  Business  Plan  to  indicate  that  any  of 
those  three  components  were  designed  to  replace  legacy  systems  resident  in 
the  combatant  commanders  or  Services.  It  is  theoretically  possible,  but 
unlikely,  for  RAS-OT  to  replace  a  Service  tool.  Service  tools  query  their 
respective  SORTS  databases  that  are  of  greater  breadth  compared  to  GSORTS. 
That  is  to  say  that  GSORTS  data  is  a  large,  but  not  complete  representation  of 
Service  SORTS  data.  Therefore,  a  tool  designed  to  query  GSORTS  data  will  not 
necessarily  cover  the  full  range  of  a  Service  SORTS  database. 

Point  Three:  According  to  the  RAS  Business  Plan,  GSORTS(E)(I)  was  designed 
“to  replace  service  unique  input  tools  so  that  there  is  a  single  tool  that  can  be 
centrally  modified  as  new  requirements  are  added.”  (RAS  Business  Plan;  pg.  III- 
2)  Only  two  of  four  Services  are  interested  in  adopting  the  Input  Tool.  DISA  is 
developing  Input  Tools  for  the  Air  Force  and  Marine  Corps.  The  Army  and 
Navy’s  custom  input  tools  feed  portions  of  their  respective  SORTS  databases 
beyond  the  scope  of  the  RAS  Input  Tool.  The  Joint  Staff  will  not  require  the 
Army  or  Navy  to  adopt  the  RAS  Input  Tool. 

Point  Fovir.  The  description  of  the  RAS  Output  Tool  in  the  DOD  IG  draft  report 
(page  3)  is  accurate  and  captures  the  true  intent  of  the  tool.  (One 
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administrative  correction:  That  paragraph  incorrectly  lists  the  acronym  for 
Global  Status  of  Resources  and  Training  System  as  SORTS.  The  correct  term 
is  GSORTS.  SORTS  and  GSORTS  are  not  synonymous.) 

The  DOD  IG  audit  limited  itself  to  a  historical  review  and  failed  to  consider 
current  and  future  development  plans  for  RAS-OT.  The  current  timeline  and 
reengineering  effort  place  RAS-OT  on  a  secure  development  schedule  that 
aggressively  addresses  previously  discovered  performance  shortfalls  and 
involves  representatives  from  the  readiness  user  community  where  appropriate. 

DOD  IG  Findings: 

Finding  I.  (Page  4,  1st  paragraph,  1st  line)  ^DISA  and  the  Joint  Staff  have  not 
provided  sufficient  assurance  that  the  RAS  Ou^mt  Tool  will  satisfy  user 
requirements,"  The  Joint  Staff  nonconcurs.  The  Joint  Staff  amassed  a  total  of 
734  requirements  for  RAS-OT  from  combatant  commands  and  Services.  The 
list  is  constantly  validated  and  updated.  These  requirements  are  well 
documented  and  briefed  as  necessary  to  the  Global  Command  and  Control 
Readiness  Working  Group  (GCC  RWG)  which  is  composed  of  action  officers 
from  OSD,  the  Joint  Staff,  and  the  Services;  the  Flag  Officer/General  Officer 
Executive  Steering  Committee  (ESC);  and  annually  to  the  GCC  RWG  User 
Review  Panel  which  represents  the  worldwide  readiness  community  including 
combatant  commands  and  Defense  agencies. 

The  current  breakdown  of  RAS-OT  requirements  is  as  follows:  395 
requirements  will  be  incorporated  into  the  first  global  release  of  RAS-OT, 
version  4.0.  These  395  requirements  were  selected  for  inclusion  into  version 
4.0  because  they  are  requirements  common  to  most  users  and  are 
technologically  feasible  within  the  current  GSORTS  architecture.  These 
requirements  are  maintained  by  the  J-3  Readiness  Division  and  will  not  be 
vetted  with  combatant  commanders  or  Services  again.  To  do  so  would 
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jeopardize  the  development  schedule.  This  is  within  the  responsibilities  of  The 
Joint  Staff  as  acknowledged  by  the  DOD IG.  (Page  1,  paragraph  4,  1st  line) 

"The  Jb^  Staff...  as  the  project  sponsor  for  cdl  GCCS  cg)plications,  represents  the 
needs  of  the  users.  Throughaut  the  requirements  validation  and  approval 
process.  The  Joint  Staff  identifies  and  priontizes  functional  requirements  that 
must  be  satisfied  by  GCCS.* 

The  last  formal,  comprehensive  review  of  the  total  requirement  database 
occurred  in  June  01  in  support  of  the  Functional  Qualification  Test.  The 
results  of  the  test  were  docinnented  and  briefed  to  the  ESC  in  July  01 .  The 
results  of  the  test  and  recommendations  to  the  ESC  were  the  vehicle  to  solidify 
requirements  for  RAS-OT  version  4.0,  and  to  recommend  the  current  timeline. 
That  timeline  allows  for  a  major  reengineering  effort  to  produce  a  quality 
product,  which  is  designed  to  user  specifications.  The  ESC  approved  the  set  of 
requirements  that  defme  RAS-OT  version  4.0  and  the  new  developmental  time 
line. 

Further,  290  of  the  734  requirements  are  categorized  as  "future** 
requirements  because  they  are  Service  unique  or  are  not  within  the  technical 
capability  of  RAS-OT  version  4.0  or  are  not  possible  within  the  current 
GSORTS  database  architecture.  These  requirements  will  be  vetted  with 
combatant  commands,  combat  support  agencies,  and  Services  during  Phase  III 
of  the  RAS  Business  Plan  which  is  also  Phase  III  of  the  revised  RAS-OT 
schedule  which  was  approved  by  the  ESC  in  July  01 .  Additional  new 
requirements  will  be  considered  for  inclusion  into  the  master  list. 

The  remaining  49  of  734  requirements  were  removed  from  the  master 
database  because  they  were  either  duplicates  of  other  valid  requirements, 
impossible  to  measure,  or  beyond  the  technological  capability  of  foreseeable 
versions  of  GSORTS  or  RAS-OT,  Requirements  removed  from  the  master 
database  are  maintained  in  the  overall  RAS-OT  historical  archive. 

Finding  2.  (Page  4,  1st  paragraph  2d  line)  .  .  DISA  and  the  Joint  Staff  failed 
to  meet  RAS  Output  Tool  development  and  fielding  milestones.  This  condition 
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occurred  because  DISA  and  the  Joint  Staff  did  not  involve  the  users  throughout 
the  development  process,  implement  adequate  testing  procedures,  fully  document 
critical  decisions,  and  establish  effective  performance  measurements  to  monitor 
development 

1 .  ^DISA  and  the  Joint  Staff  failed  to  meet  RAS  Output  Tool  development 
and  fielding  milestones.’"  The  Joint  Staff  concurs*  The  fielding  of  RAS-OT  has 
been  delayed  multiple  times.  See  chart  on  page  10  of  the  IG  report. 

2.  “This  occurred  because  DISA  and  The  Joint  Staff  did  not  involve  the  user 
community  throughout  the  development  process.’"  The  Joint  Staff  nonconcurs. 
Users  have  been,  and  will  be,  involved  at  every  appropriate  step  in  the 
developmental  process.  As  described  earlier,  users  from  the  readiness 
community  did  define  the  requirements.  After  that  point,  user  involvement  is 
limited  until  a  production  representative  s}rstem  is  developed.  Users  do  not 
participate  in  the  developmental  phase  of  the  system  acquisition  process,  nor 
do  they  conduct  formal  developmental  testing.  That  is  the  sole  responsibility  of 
the  Program  Manager.  Users  participate  heavily  in  operational  testing  and 
feedback  on  “production  representative’'  systems.  That  feedback  is  invaluable 
to  the  developmental  process  and  is  insisted  upon  by  The  Joint  Staff. 

The  Joint  Staff  and  DISA  will  involve  users  in  the  Functional  Users  Test 
currently  scheduled  for  June  02.  Testing  and  development  up  until  this  point 
have  been  the  sole  responsibility  of  the  GCCS  Program  Manager,  DISA.  The 
DOD  IG  criticized  The  Joint  Staff  and  DISA  for  not  including  users  in  all 
phases  of  the  developmental  process  even  though  their  participation  would 
have  been  inappropriate  and  counterproductive.  Novices  to  systems 
development  methodology  would  likely  provide  conflicting  or  confusing 
feedback  that  would  be  more  likely  to  disrupt,  not  enhance  the  process.  The 
Joint  Staff  and  DISA  will  capitalize  on  user  input  at  every  appropriate 
opportunity. 
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3.  (Page  4,  1st  paragraph,  line  6)  “DISA  and  the  Joint  Staff /atted  to  meet 
RAS  Output  Tool  development  and  fielding  milestones.  This  oonditixm  occurred 
be&iuse  DISA  and  the  Joint  Staff  did  not.JmpleTnent  adequate  testing 
procedures  .  . .  ^  The  Joint  Staff  nonconcurs,  RAS-OT  was  tested  on  numerous 
occasions.  Scheduling  delays  were  a  direct  result  of  system  failures  discovered 
dviring  formal,  planned  testing  events. 

The  DOD  IG  statement  from  page  8,  paragraph  4,  1st  line  states  the 
issue  in  a  different  way.  That  statement  reads,  “TesUng.  DISA  and  the  Joint 
Staff  failed  to  implement  adequate  testing  procedures  to  address  user  concerns 
about  the  operational  capabilities  of  the  appUcatiorL  With  the  exception  of  U,S. 
Joint  Forces  Command,  the  Unified  Commands  and  the  Services  had  not 
participated  in  RAS  Output  Tool  testing.*  USJFCOM  was  invited  to  view  the 
work  in  progress  because  DISA  wanted  feedback  on  the  JRAMS  functionality, 
which  is  being  incorporated  into  RAS-OT.  We  received  valuable  feedback  from 
USJFCOM  and  developed  eight  new  requirements,  which  will  be  included  in 
RAS-OT  ver  4,0.  Other  USJFCOM  comments  from  page  8  of  the  DOD  IG  audit 
mirror  shortcomings  that  DISA  and  the  Joint  Staff  had  already  identified  as  a 
result  of  the  Functional  Qiialification  Testing  from  Jxme  01,  These  findings 
resulted  in  the  ESC  decision  to  reengineer  the  tool. 

(Page  8,  paragraph  4,  12th  line)  ^adequate.  Since  the  time  of  the 
preliminary  requirements  definition,  DISA  has  provided  limited  hands-on  testing 
experience,  u^hich  has  diminished  the  potential  that  the  users  unll  gain  the 
confidence  that  the  RAS  Output  Tool  uHll  satisfy  their  needs.*  USJFCOM's 
comments  are  typical  of  users  who  apply  standards  for  mature  systems,  like 
their  own  JRAMS,  against  evolving  systems  such  as  RAS-OT,  Involving  users 
too  early  in  the  process  usually,  as  it  did  in  this  case,  erodes  their  confidence 
in  the  system.  In  practice,  involving  users  too  early  has  precisely  the  opposite 
effect  recommended  by  the  DOD  IG.  Users  will  participate  in  the  Functional 
Users  Test  scheduled  for  June  02  and  at  other  appropriate  events  later  in  the 
project. 
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4.  (Page  4,  1st  paragraph,  2d  line)  *^DISA  and  the  Joint  Staff  failed  to  meet 
RAS  Output  Tool  devetopment  and  fielding  milestones.  That  condition  occurred 
because  DISA  and  the  Joint  Staff  did  not,,  fully  document  critical  decision,.  . . 
The  Joint  Staff  nonconcurs.  To  properly  address  this  assertion  the  Joint  Staff 
requests  specific  examples  of  critical  decisions  that  were  not  documented  and 
the  reference  for  determining  precisely  how  those  decisions  should  have  been 
documented.  The  chart  on  page  10  of  the  DOD  IG  draft  identified  the  four 
primaiy  critical  decisions  that  authorized  delays  in  the  RAS-OT  projected 
deliveiy  dates.  That  chart  is  titled,  “RAS  Output  Tool  Fielding  Delays.* 

5.  (Page  4,  1st  paragraph,  7th  line)  “D/SA  and  the  Joint  Staff  failed  to 
meet  RAS  Output  Tool  development  and  fielding  milestones.  That  condition 
occurred  beamse..,DISA  and  the  Joint  Staff  did  not... establish  effective 
performance  measurements  to  monitor  development  The  DOD  IG  further  states 
on  page  9,  6th  paragraph,  of  the  draft,  ^Pefformance  Measures.  DISA  and  the 
Joint  Staff  did  not  develop  an  adequate  mechanism  that  would  measure 
performance  against  standards.  The  Joint  Staff  nonconcurs,  DISA  and  the 
Joint  Staff  did  establish  effective  performance  measurements  to  monitor 
development  and  measure  success,  or  failure,  against  a  specific  set  of  criteria. 
The  Requirements  Traceability  Matrix  (RTM),  a  component  of  the  System 
Requirements  Specifications  (SRS),  methodically  compares  specific  system 
performance  expectations  to  documented  ssrstem  requirements.  Every  single 
approved  requirement  slated  for  RAS-OT  version  4.0  has  a  documented  test 
procedure  and  an  expected  outcome. 

Prom  a  GCCS  perspective,  the  Joint  Staff  is  constructing  an  Operational 
Requirements  Dociament  for  GCCS  that  incorporates  GSORTS  and  RAS-OT 
requirements  including  threshold  and  objective  parameters.  This  is  a  Joint 
Staff  action  and  not  the  responsibilily  of  DISA. 
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Finding  3.  (Page  4,  2d  paragraph)  ‘In  addition,  delays  in  fielding  made  it 
necessary  for  the  Unified  Commands  and  Services  to  develop  new  appUcatians  or 
mxxtijy  existing  applications  that  would  satisfy  their  unique  r&piireTnents  for 
performing  readiness  analysis.’*  The  Joint  Staff  nonconciirs.  This  assertion 
draws  an  incorrect  corollary  between  RAS-OT  delays  and  development  of 
combatant  command  or  Service  tools. 

The  three  analytical  systems  outlined  in  the  original  concept  for  RAS 
Phase  II:  AJMRR,  GS0RTS(E)(0),  and  JEMP  III,  were  never  intended  to  replace 
combatant  command  or  Service  specific  tools.  The  GSORTS(E)(I)  was  designed 
to  replace  Service  input  tools,  but  the  input  tool  is  not  an  application  for 
performing  analysis. 

The  first  opportunity  to  cause  combatant  commands  or  Services  to 
develop  or  modify  their  applications  because  of  a  RAS  delay  would  have  been 
with  the  first  slip  of  the  RAS  development  schedule,  November  1999  (as  per  the 
chart  on  page  10  of  the  DOD  IG  draft  report).  However,  combatant  command 
and  Service  readiness  applications  predate  November  1999.  USJFCOM's 
JRAMS  began  development  in  1995  and  JRAMS  Classic  was  fielded  in  1997. 
The  Army's  readiness  management  system,  ARMS,  has  been  in  use  since  1989. 
The  Navy's  Type  Commanders'  Readiness  Management  System  (TRMS) 
predates  1999,  the  Marine  Corps'  Global  Online  Marine  Edit  and  Reporting 
System  (GOMERS)  predates  1999.  HQ  Air  Force  is  developing  a  new  tool  to 
tackle  the  issue  of  "predictive  readiness."  This  concept  is  beyond  the  scope  of 
RAS-OT  or  any  other  GCCS  application.  It  is  therefore  incorrect  to  blame 
delays  in  RAS-OT  for  the  Air  Force  decision  to  create  this  capability.  The  J-3, 
Readiness  Division,  knows  of  no  combatant  command  or  Service  who  based  a 
readiness  development  decision  on  RAS-OT  schedule  delays. 

Finding  4.  User  Requirements 

1.  The  DOD  IG  surve3^  the  four  Services  and  six  unified  commands 
but  foimd  a  lack  of  "community  awareness."  This  is  not  surprising  considering 
personnel  turnover  rates  compared  to  the  RAS  developmental  timeUnc.  Simply 
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stated^  most  cxirrent  combatant  command  and  Service  representatives  were  not 
in  their  current  job  at  the  time  the  RAS  Business  Plan  was  staffed  or  during 
the  RAS-OT  requirement  development  process. 

The  DOD  IG  draft  report  states,  (page  6,  3d  paragraph,  3d  line) 
"requirements.  We  surveyed  each  of  the  four  Services  and  six  of  the  nine  Unified 
Commands  to  assess  users  awareness  of  and  participation  in  the  RAS  Output 
Tool  requirement  identification  and  development  The  U.S.  European  Command, 
the  U.S.  Pacific  Command,  the  U.S.  ^redal  Operations  Command,  and  the  Air 
Force  stated  they  had  limited  knowledge  of  the  RAS  Ou^mt  Tool  In  addition,  the 
U.S.  Central  Command,  the  U.S.  Southern  Command,  and  the  Navy  all  stated 
that  they  had  no  knowledge  of  the  RAS  Output  Tool  ” 

In  sequence:  the  US  European  Command  has  a  functioning  version  of 
the  RAS-OT  in  the  readiness  division  of  their  headquarters.  This  installation 
was  coordinated  in  the  first  quarter  of  FY  02  and  completed  in  January  2002. 
USPACOM  has  been  briefed  on  RAS-OT  by  the  Joint  Staff,  and  their  J5 
planners  expressed  interest  in  acquiring  the  tool.  US  Special  Operations 
Command  not  only  has  extensive  knowledge  of  RAS-OT,  it  is  schedtiled  to 
receive  a  production  representative  version  of  the  software  in  May  2002. 
USSOCOM  has  also  agreed  to  participate  in  the  next  test,  June  2002. 

Air  Force  submitted  a  significant  number  of  RAS-OT  requirements,  is  a 
member  of  the  Global  Command  and  Control  Readiness  Working  Group  (GCC 
RWG),  and  has  been  working  with  DISA  APC  19  to  develop  an  Air  Force  version 
of  the  RAS  Input  Tool.  The  Air  Force  has  production  representative  RAS-OT 
software  loaded  on  at  least  three  GCCS  work  stations  and  will  participate  in  the 
next  formal  RAS-OT  test.  Like  the  Air  Force,  Navy  is  a  member  of  the  GCC 
RWG  and  has  a  working  version  of  RAS-OT  in  their  headquarters  readiness 
office. 

In  addition  to  the  above-mentioned  commands,  USTRANSCOM  received  a 
production  representative  version  of  RAS-OT  and  has  agreed  to  participate  in 
future  testing.  HQ  Army,  HQ  USMC,  and  J-3,  DDGO  Readiness,  all  have  the 
latest  versions  of  RAS-OT  working  on  GCCS  workstations.  The  DOD  IG,  in 
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many  cases,  may  have  failed  to  interview  the  right  people  at  several  combatant 
commands  and  two  Services. 

2.  (Page  6,  3d  paragraph,  i  1th  line)  .  .  the  U.S.  Joint  Forces  CoTTimond 
stated  that  the  RAS  Output  Tool  did  not  provide  Joint  Readiness  Automated 
Management  System  (JRAMS)  functionality  and  performed  significantly  slower 
than  JRAMS.  *'  The  RAS  Output  Tool  does  contain  value  added  JRAMS 
functionality,  but  not  every  aspect  of  JRAMS  is  included  in  RAS-OT.  That  is  by 
design.  It  is  important  to  understand  that  JRAMS  and  I?AS-OT,  initially,  were 
both  developed  by  the  same  contractor,  SAIC.  SAIC  was  replaced  because  they 
could  not  satisfy  the  full  range  of  RAS-OT  user  reqxnrements.  In  short,  RAS-OT 
started  out  as  a  tool  designed  very  much  along  the  lines  of  JRAMS  but  could 
not  grow  to  satisfy  the  RAS-OT  user  requirements. 

USJrcOM  is  not  the  only  RAS-OT  customer.  The  DOD  IG  criticizes  the 
fact  that  RAS-OT  is  not  like  JRAMS.  The  DOD  IG  implies  that  JRAMS  could 
have  been  a  better  candidate  than  RAS-OT.  At  the  same  time,  the  DOD  IG 
criticizes,  the  lack  of  user  involvement  in  requirements  generation  for  RAS-OT. 
JRAMS  was  not  built  to  combatant  command  and  Service  specifications. 

JRAMS  was  built  to  USJFCOM  specifications.  JRAMS  could  not  satisfy  the 
global  readiness  commtmities'  requirements  in  1997  nor  could  the  application's 
architecture  be  modified  to  adequately  incorporate  user  requirements  by  1999. 

Initial  acceptance  of  the  ACTD  product  was  not  sufficiently  tested  to 
scope  the  project.  JRAMS  code  was  not  compatible  with  the  GCCS  architecture 
(Defense  Information  Infrastructure  Common  Operating  Environment  (Dll 
COE)).  Secondly,  the  contractor,  SAIC,  failed  to  meet  performance  standards 
and  was  replaced  by  AB  Floyd.  This  introduced  significant  delays  and 
restarted  the  development  process. 

JRAMS  is  an  effective  application.  There  are  features  in  JRAMS  whose 
incorporation  make  RAS-OT  a  better  product,  but  a  direct  comparison  of  the 
two  applications  is  not  appropriate  in  this  case.  RAS-OT  is  slower  than 
JRAMS,  in  part,  because  it  processes  more  information,  serves  a  ^obal 
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population  of  users,  and  satisfies  a  much  larger  collection  of  user-defined 
requirements.  However,  the  fundamental  lack  of  speed  was  a  major  factor  in 
the  recommendation  to  the  ESC  in  July  01  to  reengineer  the  took  That 
decision  has  produced  tremendous  results.  RAS-OT  is  now  much  faster  and 
the  quality  of  the  entire  application  has  improved  dramatically  from  July  2001 
to  present.  Recent  testing  by  J-3  Readiness  and  DISA  (Februaiy  02)  indicates 
RAS-OT  will  now  pass  the  speed  requirements  dictated  in  the  formal 
requirements  list. 

Additionally,  JRAMS  was  not  built  to  comply  with  GCCS  architecture. 
JRAMS  architecture  is  not  compatible  with  security  and  validation  protocols  for 
GCCS.  As  stated  on  page  1  of  the  DOD  IG  audit,  (Page  1,  1st  paragraph,  5th 
lme)\  .  .  GCCS  consists  of  interoperable  hardware,  software,  common 
procedures,  standards,  and  interfaces  that  comprise  an  ^^operational 
architecture.^  JRAMS  was  not  designed  to  be  GCCS  compatible  or  DII  COE 
compliant. 

Finally,  JRAMS  is  USJFCOM's  application  of  choice  because  they  built  it 
to  their  specifications.  USJFCOM  receives  funding  annually  from  DISA  to 
maintain  JRAMS.  Success  of  the  RAS-OT  may  be  perceived  by  USJFCOM  as  a 
potential  threat  to  that  annual  funding  stream  (i.e.,  USJFCOM  may  not 
necessarily  represent  the  JRAMS/ RAS-OT  issue  from  an  unbiased  position). 
Recommend  that,  in  this  case,  the  DOD  IG  consider  the  objectivity  of  the 
source  for  this  issue. 

Finding  5^  RA8  Businmas  Plan  comments  ipa,  71.  Issues  involving  validation 
of  requirements  have  been  discussed.  One  must  be  careful  to  draw  2002 
conclusions  from  a  1998  business  plan.  It  was  vetted  by  working  groups 
during  its  development.  The  Services  all  provided  comments  that  were 
incorporated  in  the  final  product.  In  the  end,  each  Service  concurred. 

1.  SeeDef  Comments.  The  DOD  IG  cites  a  specific  SecDef  “substantive 
comment"  to  be  included  in  the  revision  of  the  Business  Plan.  (Page  7,  5th 
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paragraph)  ^’The  key  to  the  utility  and  effectiveness  of  the  RAS  is  the 
involvement  of  the  CINCs,  .,and  other  users  in  the  development  process,  Eariy 
involvement  will  improve  understanding  of  the  potential  of  RAS  and  will  lead  to 
the  identification  of  new  requirements,’^  The  DOD  IG  uses  this  quote  out  of 
context.  The  above  stated  paragraph  is  specifically  directed  to  be  added  to 
Chapter  IV  at  the  bottom  of  page  IV-2.  This  section  addresses  Phase  III  of  the 
RAS  Business  Plan.  “Integration  of  New  Indicators.  Defined."  This  chapter  is 
dedicated  to  a  phase  of  RAS  development  that  has  not  yet  begun.  According  to 
the  Business  Plan,  “Phase  III  involves  the  future  planned  integration  of  selected 
portions  of  the  Joint  Total  Asset  Visibility  (JTAV)  system,  Joint  Personnel  Asset 
Visibility  (JPAV)  system,  and  Defense  Integrated  Military  Human  Resources 
System  (DIMHRS)  leading  indicators  into  RAS."*  This  capability  is  too  far  down 
the  road  to  conceptualize  yet  alone  make  deliberate  plans.  Some  of  the 
systems  described  in  Phase  III  do  not  even  exist  today.  The  SecDef  comment 
used  by  the  DOD  IG  to  substantiate  their  case  for  requirements  development 
does  not  apply  to  the  Phase  of  RAS-OT  development  addressed  by  this  audit. 
The  Joint  Staff  requests  the  SecDef  quote  be  removed  from  this  audit. 

2.  Army  Comments.  The  DOD  IG  quote  of  the  Army  comment  to  the 
RAS  Business  plan  is  inaccurate.  The  DOD  IG  report  quotes  the  Army 
comment  as  follows,  (Page  7,  6th  paragraph  2d  line)  . ,  However,  Army 
officials  did  not  beliei/e  that  DISA  and  the  Joint  Staff  had  performed  an  adequate 
mission  needs  analysis  or  validated  the  requirement  for  a  readiness  assessment 
application  as  comprehensii^  as  RAS.’*  The  Army  comment  did  not  mention  the 
GSORTS  (RAS)  Output  Tool.  The  Army  was  commenting  on  the  full  RAS 
business  plan.  The  comment  was  taken  out  of  context  and  misrepresents 
Army  intent.  One  has  to  only  read  the  two  paragraphs  that  follow  to  get  the 
true  Army  purpose  behind  their  comment.  The  Army  was  not  worried  that  the 
tool  would  not  meet  their  needs,  they  were  worried  that  the  RAS  Business  Plan 
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was  too  invasive^  perhaps  intruding  on  the  Services'  Title  X  responsibilities. 

The  subsequent  paragraphs  from  the  Army  statement  read  as  such; 

^The  recurring  concepts  supporting  HAS  are  ^‘commanders  at  aU 
teoels"  and  ‘near  real  time*  readiness  information.  Army  believes  a  more 
thorough  mission  needs  analysis  is  needed  to  determine  more  precisely 
the  in/ormatton  timeliness  needs  at  each  level  of  command,  and  to  target 
that  information  to  the  needs  of  the  reorient.  For  instance,  policy  makers 
do  not  need  minute  details  or  24  hour  readiness  updates  to  carry  out 
po/iq/-7nafctn^  responsibilities.  Additionally,  in  the  world  of  deliberate 
planning,  current  readiness  data  is  not  useful  Force  sourcing  decisions 
drive  resourcing  which  ultimately  determines  readiness.  Only  in  the 
souncihp  of  contingency  operation  force  requirements  is  current  readiness  a 
factor,... 

RAS  seems  to  be  driven  more  by  the  fact  that  the  technology  to 
achieve  a  directed  telescope  to  the  forward  foxhole  will  soon  be  available 
than  by  any  validated  need.  Army  acknowledges  that  RAS  consists  of  a 
grouping  of  systems  and  software  which  are  already  in  some  state  of 
development  or  fielding,  many  of  which  are  already  funded  and  are 
intended  to  be  resident  in  the  GCCS/GCSS  suites.  However,  Army 
believes  that  the  true  cost  of  a  comprehensive  RAS  has  not  been 
determined.  Finally,  Army  is  concerned  that  the  potential  for 
micromariagement  inherent  in  RAS  represents  an  intentional  or 
unintentional  intrusion  into  the  Services*  title  X  responsibilities  to  provide 
ready  forces  to  the  combatant  commanders.* 

The  DOD  IG  snapshots  of  RAS  Business  Plan  comments  from  the  Navy, 
Air  Force,  and  Marine  Corps  are  more  accurate  but  still  represent  the  RAS 
Business  Plan  in  its  entirety,  not  necessarily  the  GS0RTS(E)(0)  now  known  as 
RAS-OT.  Remember,  the  objective  of  the  DOD  IG  audit:  ‘The  specific  objective 
for  this  segment  of  the  audit  was  to  assess  management  and  oversight  of  the 
Readiness  Assessment  System  Output  Tool*  (Page  3,  Objectives,  4th  line)  The 
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Joint  Staff  requests  the  DOD  IG  recognize  this  in  the  audit  or  delete  Service 
comments  that  do  not  specifically  apply  to  RAS-OT. 

Search  for  Candidate  Solntions  and  Analysis  of  Alternatives*  (Page  9,  3d 
paragraph,  2d  line)  .  .DISA  amid  not  doaxment  that  they  a}Tiducted  an 
appropriate  search  for  candidate  qppficottons/or  the  RAS  Output  TooL^  (Page  9, 
paragraph  4)  and  “Existing  Service  or  Untied  Ck>mmand  tools  might  have  met 
RAS  Output  Tool  requirements,  but  the  Joint  Staff  and  DISA  could  not  document 
that  they  conducted  an  c^propnate  search /or  candidate  applications  for  the  RAS 
Output  Tool ...  *  The  Joint  Staff  concurs  in  comment.  An  extensive  search  of 
our  records  could  not  provide  the  requested  documentation.  The  search  for 
candidate  solutions  and  analysis  of  alternatives  was  complete  by  the  time  DISA 
took  over  the  project  from  DARPA.  By  the  time  the  business  plan  was 
approved  by  the  Secretary  of  Defense,  the  decisions  were  made  and  DISA 
proceeded  to  build  upon  the  work  begun  by  SAIC. 

The  20  March  02  cover  letter  to  the  DOD  IQ  Drctft  Audit  directs  the  to 
^comment  on  the  materiat  control  wetUenees  discussed  tn  Appendix  A,^ 

(page  1 7,  last  paragraph)  “We  identified  material  Tnanagement  control 
tueaknesses  within  the  Joint  StajQTand  DISA  as  defined  by  DoD  Instruction 
501 0.40.  ^)ecfically,  management  controls  of  the  Joint  Staff  for  GCCS  oversight 
and  requirements  development  did  not  exist“  The  Management  Control 
Program  Review  from  the  DOD  IG  Audit  Appendix  A  refers  to  GCCS-level 
policies  and  procedures.  Recommendations  in  this  area  specifically  address 
improving  GCCS  management  and  development  for  new  applications.  Both 
GCCS  Management  Structure  and  the  GCCS  Operational  Requirements 
Document  are  currently  under  revision  and  is  a  priority  within  the  Director  for 
Operations,  J-3. 
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POD  IG  RamtiifMiittdatioiM  <Pg,  14\ 

The  DOD  IG  reconimends  the  Director  for  Operations,  Joint  Staff: 

1  .a.  “Revalidate  the  or^^mol  Ghbal  Command  and  Control  System 
Readiness  Assessment  System  Output  Tool  requirements  with  current  users 
within  the  Unified  Commands,  Services,  and  Defense  Agencies  and  ensure  that 
Juture  Ghbal  Command  and  Control  System  user  requirements  are  implemented 
as  validated."  The  Joint  Staff  conc^lrs  in  the  intent  of  the  recoimnendation  but 
would  modify  the  precise  execution  of  the  task.  The  total  list  of  RAS-OT 
requirements  numbers  734.  The  requirements  supporting  the  current  software 
version  under  construction  number  395.  Those  requirements  are  locked-in 
and  will  not  change.  This  is  a  developmental  necessity.  Software  engineers 
cannot  hit  a  moving  target  resulting  from  changing  requirements.  J-3, 
Readiness,  and  DISA  will  review  the  395  requirements  again  in  preparation  for 
the  June  02  test.  Of  note;  as  of  15  April  02,  DISA  has  satisfied  370  of  395 
requirements.  RAS-OT  4,0  is  very  much  on  track.  The  Joint  Staff  secs  no 
reason  why  personnel  turnover  among  the  combatant  commands  or  Services 
justifies  revalidating  user  requirements  for  the  software  version  imder 
development.  To  do  so  wovild  cause  a  significant  delay  in  the  developmental 
process.  Any  review  that  would  impede  or  delay  the  development  process  of 
RAS-OT  version  4.0  is  unacceptable  to  the  Joint  Staff, 

The  remaining  290  (plus)  user  requirements,  and  others  that  will  be 
collected  over  the  next  year,  are  earmarked  for  RAS-OT  versions  beyond  4.0. 

All  combatant  commands,  combat  support  agencies,  and  Services  will  be 
invited  to  review  those  requirements.  This  review  will  take  place  shortly  after 
the  June  02  E^AS-OT  test.  This  will  allow  DISA  and  the  Joint  Staff  to  publish 
the  results  of  the  test,  publish  the  set  of  requirements  supporting  RAS-OT 
version  4.0,  and  ask  the  user  community  to  review  and  comment  on  the 
remaining  user  requirements  that  coiild  potentially  go  into  future  RAS-OT 
software  builds. 
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1  .b.  "For  future  GCCS  Readiness  applications,  search  Government  and 
commerdat  sources  for  applications  that  will  satisfy  requirements  and  document 
the  results.*  The  Joint  Staff  nonconcurs.  Current  OSD  initiatives  to 
conceptualize  and  develop  an  improved  readiness  reporting  system  for  DOD  are 
fulfilling  this  requirement.  OSD  recently  looked  at  current  readiness 
applications  DOD-wide  to  support  their  research  effort.  A  similar  effort  by  the 
J-3  would  be  redundant  and  a  waste  of  valuable  time  and  money.  The  DOD  IG 
should  be  aware  of  the  OSD  effort  and  instead  recommend  OSD  publish  their 
findings  to  benefit  the  entire  DOD  readiness  communify.  The  Joint  Staff  will 
make  sure  future  studies  of  GCCS  candidate  applications  are  thoroughly 
documented  and  archived  for  historical  purposes. 

1. c.  “Develop performance  standards  that,  at  a  minimum,  provide  a  way 
of  assessing  the  performance,  cost,  schedule,  technical  quality  and  reliability, 
and  user  satisfaction  and  formally  report  on  the  progress  of  the  Readiness 

Assessment  System  Output  Tool  in  meeting  those  standards.*  The  Joint  Staff 
concurs,  but  will  not  delay  the  RAS-OT  developmental  effort  during  this 
process. 

Joint  stair  comment  on  one  DoD  IG  fecommendatlon  for  D18A  (Page  15) 

2.  a.  The  DOD  IG  recommends  DISA  (Page  15,  paragraph  2. a.)  “Pause 
development  of  the  Readiness  Assessment  System  Output  Tool  until  the  Joint 
Staff  fully  addresses  requirements  development,  user  satisfaction,  program 
documentation,  and  performance  measurement  *  The  Joint  Staff  recommends 
against  this  course  of  action.  The  rate  of  the  developmental  progress  for  RAS- 
OT  has  never  been  greater  than  it  is  right  now.  Working  prototypes  of  the  RAS- 
OT  are  resident  in  the  J-3,  Readiness  Division  office,  HQ  US  Army,  HQ  US  Air 
Force,  HQ  USMC,  HQ  Navy,  USEUCOM,  USTRANSCOM,  and  USJFCOM,  and 
will  soon  be  installed  at  the  USSOCOM.  Initial  response  to  the  prototype  is 
very  favorable.  The  first  test  of  a  production  representative  system  that  will 
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employ  users  from  the  readiness  community  including  some  combatant 
commaxids,  all  Services,  and  at  least  one  combat  support  agency  is  scheduled 
for  June  02.  A  “pause’'  would  further  exacerbate  the  same  condition  that  has 
brought  the  most  negative  scrutiny,  more  scheduling  delays.  Previous  delays 
were  caused  by  failures  of  the  software  to  meet  user  requirements.  A  delay  for 
the  sake  of  staff  analysis  would  be  inappropriate.  The  recommendations 
described  by  the  DOD  IG  can  be  accomplished  as  a  parallel  effort. 
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DEFENSE  INFORMATION  SYSTEMS  AGENCY 

701  S.  COURTHOUSE  ROAD 
ARLINGTON,  VIRGINIA  22204^2199 


REFEftTO:  INSPECTOR  GENEEIAL  (IG) 


3  May  2002 


MEMORANDUM  FOR  DOD  Inspector  General 

SUBJECT:  Audit  Report  on  Global  Command  Control  System 

Readiness  Assessment  System  Output  Tool 
(Project  No.  D2000LG-0101.01) 


1 .  The  enclosed  document  provides  a  response  from  the  Defense 
Information  Systems  Agency  on  the  subject  DOD  IG  Draft  Report. 

2.  If  you  have  any  questions,  please  call  Teddie  Lou  Steiner, 
Audit  Liaison,  at  (703)  607-6316  or  Liz  Lippmann,  Audit  Liaison, 
at  (703)  607-6306. 

FOR  THE  DIRECTOR: 
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DKVEKSB  XNfCSMKTIOH  SYSTEMS  JU3BMCY  (OISA)  BESPONSB  TO 
DoCfIG  inUkR  ADDIT  ASSORT 

''aLOBAL  and  c^sitaoi.  system 

BSAD1SS8S  ASSSSSMEEIT  SYSTEM  OOTFOT  TOOL" 
Project  No.  D20001LO-0101.01  of  20  Kftroh  2002 


IntroduotioD 

•  In  addition  to  providing  DISA  concurrence/  nonconcurrence/ 
comments  to  the  DoDIG's  recommendations  for  DISA,  DISA  is 
also  providing  informational  comments  on  the 
Recommendations  for  the  Director  for  Operations,  Joint 
Staff,  where  the  recommendation  and  appropriate  action 
include  DISA  efforts. 

•  Comments  regarding  the  discussion  portion  of  the  draft 
report  are  attached  as  background  information, 

•  Throughout,  direct  quotes  from  the  DOD  IG  report  are 
presented  in  italics  with  page  numbers  from  the  draft 
audit.  Other  quotes  are  listed  in  italics  as  well  with 
appropriate  citations. 


DOD  IG  ReooiPBiendatione 

1.  Recommendations  for  the  Director  for  Operations,  Joint  Staff: 

a.  ''Revalidate  the  original  GCCS  RAS  OT  requirements  with 
current  users  within  the  Unified  Commands ^  Services,  and 
Defense  Agencies  and  ensure  that  future  GCCS  user 
requirements  are  validated  and  implemented  in  accordance 
with  current  guidance. 

Requirements  were  recently  reviewed  at  the  direction  of  JS  J3 
DDGO-R  and  will  be  reviewed  again  as  part  of  ongoing 
development.  The  full  set  of  user  requirements  were  revalidated 
in  May  2001,  by  the  JS  J3  DDGO-R  in  preparation  for  the 
Functional  Qualification  Testing  in  June  2001.  The  results  were 
briefed  to  the  RAS  ESC  in  July  2001.  The  Joint  Staff  and  DISA 
will  review  the  full  user  requirements  list  again  in  May  2002  in 
preparation  for  RAS  OT  Functional  User  Testing  scheduled  for 
June  2002.  Joint  Staff  has  indicated  they  will  review  user 
requirements  slated  for  RAS  OT  versions  beyond  4.0  as  well  as 
any  additional  requirements  collected.  Joint  Staff  has 
indicated  this  effort  will  include  all  Combatant  Command,  Combat 
Support  Agency,  and  Service  readiness  divisions.  To  ensure 
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clarity,  delate  the  phrase  '^GCCS  user  requirements"  and  replace 
with  "RAS  OT  user  requirements." 

b.  "For  Future  GCCS  Readiness  applications,  search 
Government  and  commercial  sources  for  applications  that  will 
satisfy  requirements  and  document  the  results  of  those 
searches.'^ 

Current  OSD  initiatives  to  conceptualize  and- develop  an  improved 
readiness  reporting  system  for  DOD  are  fulfilling  this 
requirement.  The  Joint  Staff  and  DISA  are  involved  in  this 
effort.  The  Joint  Staff  is  also  involved  in  a  similar  process 
as  part  of  the  GCCS  ORD  development.  Additional  efforts  in  this 
direction  would  be  redundant. 

c.  "Develop  performance  standards  that,  at  a  minimum, 
provide  a  way  of  assessing  the  per Forma nee,  costf  schedule, 
technical  quality  and  reliability,  and  user  satisfaction  and 
formally  report  on  the  progress  of  the  HAS  (OT)  in  meeting  those 
standards." 

It  is  the  responsibility  of  the  Joint  Staff,  as  the  Requirements 
Authority,  to  identify,  validate,  and  prioritize  GCCS 
requirements,  as  well  as  provide  objective  and  threshold 
functional  performance  parameters  to  DISA.  It  is  DISA's 
responsibility,  as  the  GCCS  Program  Manager,  to  build  a  plan  for 
the  development,  integration,  testing,  accreditation,  and 
fielding  of  RAS  OT  as  part  of  the  larger  GCCS  program.  It  is 
the  responsibility  of  the  Joint  Staff  to  represent  the  user 
community  in  the  assessment  of  how  well  the  developed 
functionality  meets  the  validated  operational  requirements 
(based  upon  established  threshold  performance  parameters)  and 
user  satisfaction  with  developed  functionality.  However,  it  is 
the  Program  Manager' s  responsibility  to  assess  cost,  schedule 
and  performance  and  report  in  accordance  with  DoD  5000. 2-R  at 
the  program  level  via  quarterly  Defense  Acquisition  Executive 
Summaries  to  the  acquisition  oversight  authority.  In  this  case, 
DISA  provides  the  Program  Manager  (GCCS)  and  ASD  (C3I)  is  the 
Milestone  Decision  Authority. 

2,  Recommendations  for  the  Director,  DISA: 

a.  "Pause  development  of  the  RAS  (OT)  until  the  Joint  StaFF 
Fully  addresses  requirements  development,  user 
satisfaction,  program  documentation,  and  performance 
measurement." 
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PISA  nonconcurs.  This  was  already  essentially  performed  during 
the  spring  and  early  summer  of  i 001^ under  the  auspices  of  the 
RAS  ESC.  Based  upon  a  full  review,  and  validation  of 
requirements,  a  thorough  Functional  Qualification  Test,  and 
feedbacJc  from  the  user  community,  the  RAS  ESC  directed  DISA  in 
July  2001  to  reengineer  the  RAS  OT.  Beginning  with  the 
installation  in  October  2001,  pre-release,  functioning  versions 
of  the  RAS  OT  are  resident  at  JS  J3  DDGO-R,  JFCOM,  EUCOM, 
TRANSCOM,  HQ  US  Navy,  HQ  US  Air  Force,  HQ  US  Army,  HQ  OSMC, 
MARFORIANT  and  MARFORRES.  The  first  test  of  an  operationally 
representative  system  involving  users  from  Combatant  Commands 
and  Services  is  scheduled  for  June  2002.  At  this  point  another 
development  pause  would  further  exacerbate  the  same  condition 
that  has  brought  the  most  negative  scrutiny,  more  scheduling 
delays.  Previous  delays  were  caused  by  failures  of  the  software 
to  meet  user  requirements.  A  delay  for  the  sake  of  staff 
analysis  would  be  inappropriate,  costly,  and  counterproductive 
to  delivering  capability  to  the  user  community. 

b.  "Develop  and  document  cost  and  feasibility  analyses  of 
solutions  that  are  identified  by  the  Joint  Staff, 

DISA  concurs.  DISA  has  a  documented  process  in  the  GCCS 
Evolutionary  Acquisition  Strategy  for  development  of  cost  and 
feasibility  analyses  of  candidate  solutions.  The  GCCS  Block  IV 
Implementation  Plan  (formally  known  as  the  Phase  IV  Evolutionary 
Phase  Implementation  Plan)  will  include  the  capabilities  that 
successfully  pass  this  analysis  and  show  how  they  are  to  be 
integrated  into  GCCS.  This  plan  is  anticipated  to  be  completed 
and  provided  to  the  GCCS  Integrating  Integrated  Product  Team  in 
the  summer  of  2002  and  submitted  for  GCCS  Milestone  Decision 
Authority  (MDA)  approval  in  4^*^  Quarter  FY  02. 

c.  "Survey  the  users  during  the  application  development 
process, 

DISA  concurs.  This  process  is  ongoing  with  the  current 
installation  of  a  series  of  engineering  drops  at  various 
locations  (see  response  in  section  3. a.}.  These  installations 
allow  users  a  hands-on  evaluation  thus  contributing  directly  to 
the  development  effort.  In  addition  to  the  upcoming  functional 
user's  test  in  June  2002,  users  are  encouraged  to  submit 
evaluations  at  any  time.  Also  the  Joint  Staff  is  planning  an 
additional  User  Review  Panel  to  evaluate  the  requirements  that 
have  not  been  incorporated  into  the  RAS  OT  schedule.  Additional 
requirements  for  future  versions  of  the  RAS  OT  will  also  be 
solicited. 
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d*  "Sapport  the  Joint  Staff  in  the  development  and 

application  of  concrete,  measurable,  and  qualitative 
performaiH^B  measur&nents  that  can  be  accurately  tested  on 
the  RAS  (OT) .  ' 

PISA  concurs  with  comment.  The  Joint  Staff  was  tasked  by  . 
ASD(C3I)  to  develop  a  GCCS  Operational  Requirements  Document 
(ORD) ,  which  will  contain  objective  and  threshold  performance 
parameters  that  are  measurable  and  testable.  It  is  important  to 
note  that  1)  the  responsibility  for  initiating  requirements 
rests  with  the  user,  not  the  program  manager,  and  2)  DISA  is 
supporting  the  GCCS  ORD  development  by  participation  in  the  JS- 
established  and  led  working  group.  Thus,  DISA  has  already 
completed  action  on  this  item  by  participating  in  the  GCCS  ORD 
development  effort. 

e,  "Support  the  Joint  Staff  in  developing  the  RAS  (OT)  for 
performance  standards  that^  at  a  minimum,  provide  a  way 
of  assessing  performance,  cost  schedule,  technical 
quality  and  reliability,  and  user  satisfaction  and 
formally  report  on  how  the  RAS  (OT)  is  meeting  those 
standards,/^ 

DISA  nonconcurs  with  this  recommendation  as  written  as  it 
doesn^t  recognize  the  distinction  of  the  two  separate,  yet 
complementary,  roles  of  the  Joint  Staff,  as  Requirements 
Authority,  and  DISA,  as  GCCS  Program  Manager,  nor  the  formal 
processes  already  in  place  for  program  planning  and  reporting. 

The  Joint  Staff,  in  identifying,  validating,  and  prioritizing 
GCCS  requirements,  is  responsible  for  providing  objective  and 
threshold  functional  performance  parameters  to  DISA.  It  is 
DISA's  responsibility  to  build  a  plan  for  the  entire  GCCS,  not 
just  RAS  OT,  that  trades-off  functional  performance,  schedule, 
and  cost  considerations  for  the  GCCS  system  as  a  whole. 
Reliability  is  not  one  of  these  three  primary  categories  of 
program  assessment.  It  would,  if  addressed,  be  an  operational 
requirement  for  functional  performance.  This  recommendation 
will  be  met  with  the  submission  of  the  GCCS  Block  IV 
Implementation  Plan  to  the  GCCS  Milestone  Decision  Authority  in 
4^*"  Quarter  FY  02. 

A  process  for  GCCS  user  satisfaction  assessment  is  already  in 
place  -  the  JS  J3  sponsors  periodic  web-based  surveys  and  is 
supported  by  DISA  in  the  development  and  execution  of  the 
survey. 
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Recommen¬ 
dation  3.e. 


Page  22 


Formal  reporting  £or  GCCS  performance/ schedule/cost  management 
is  done  in  with  poD  5000. 2-R,  at  the  Program  level 

via  quarterly  Defense  Acquisition  Executive  Summaries.  The 
formal  reporting  structure  does  not  retire  the  breakdown  of 
performance/schedule/cost  information  for  the  multitude  of 
applications  that  comprise  the  GCCS, 

f.  "Maintain  cojiysrehensive  records  of  milestones,  goals,  and 
key  decisions  that  doctment  RAS  (OT)  progress," 

PISA  concurs.  The  GCCS  Block  IV  Implementation  Plan  will 
document  how  RAS  OT  will  be  integrated  into  GCCS  during  Block 
IV. 

Managenent  Control  Program  Review  (page  17)  . 

"ffe  identified  material  management  control  weaknesses  within  the 
Joint  Staff  and  DISA  as  defined  by  DoD  Instruction  5010,40, 
Specifically,  management  controls  of  the  Joint  Staff  for  GCCS 
oversight  and  requirements  development  did  not  exist. 
J?ecommendations  l,a,,  l,b,,  and  X.c.,  if  implemented,  will 
Improve  GCCS  management  and  development  for  new  applications  in 
GCCS,  Also,  management  controls  in  DISA  for  GCCS  were  not 
adequate  to  ensure  that  GCCS  functionality  such  as  the  RAS 
Output  Tool  is  developed,  fielded,  and  documented  with 
requirements  that  the  users  determine.  Recommendations  2. a,, 
2,b,,  2,c,,  2,d,,  and  2,f,,  if  implemented,  will  improve  GCCS 
management  and  development  of  new  applications  in  GCCS," 

"DISA  addressed  Management  Controls  as  an  assessable  unit  and 
performed  tests  of  controls  on  areas  applicable  to  our  audit 
objectives,  DISA  officials  did  not  perform  adequate  tests  of 
controls  to  insure  that  applications  such  as  RAS  have  effective 
program  management  to  include  having  appropriately  developed 
user  requirements  and  adequate  baseline  tracking  of  program 
status," 

DISA  concurs  with  comment.  DISA  is  the  recipient  of  identified, 
validated,  and  prioritised  user  requirements.  It  is  hot  DISA'^s 
role  to  determine  if  user  requirements  are  "appropriately 
developed,"  The  program  planning,  monitoring,  and  reporting 
measures  discussed  in  response  to  the  recommendations  for  DISA, 
above,  will  address  the  management  control  concern  raised  in  the 
draft  report. 
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BACKGltOOftlD  COMHENTS 


1.  Obj«etivt»s  (png*  3). 

"Tile  specific  objective  for  this  segment  of  the  audit  was  to 
assess  management  and  oversight  of  the  Readiness  Assessment 
System  Output  Tool,'' 

PISA  concurs  with  the  stated  objective,  but  nonconcura  with  the 
draft  report's  descriptions  of  the  scope  and  purpose  of  the 
Readiness  Assessment  System  Output  Tool.  It  is  crucial  to  make 
the  distinction  between:  1)  the  original  concept  for  the 
Readiness  Assessment  System  (RAS)  as  a  total  system,  2)  the  RAS 
Output  Tool  (RAS  OT) ,  and  3)  the  collection  of  Readiness 
applications  managed  by  DISA  within  the  Global  Command  and 
Control  Program  Management  Office,  Per  the  stated  objective, 
the  findings  and  recommendations  in  this  report  should  apply  to 
the  RAS  OT,  and  not  to  the  original  concept  for  RAS,  as 
delineated  in  the  RAS  Business  Plan  dated  1996, 

According  to  the  Business  Plan,  RAS  would  have  combined  four 
independent  projects  in  Phase  II,  Those  projects  were  1) 
Automated  Joint  Monthly  Readiness  Review  (AJMRR) ;  2)  Global 
Status  of  Resources  and  Training  System  Enhanced  Output  (GSORTS 
(E) (0)),  now  commonly  referred  to  as  RAS  OT;  3)  Joint  Exercise 
Management  Program  III  (JEMP  III);  and  4)  Global  Status  of 
Resources  and  Training  System  Enhanced  Input  (GSORTS  (E) (I)). 

Point  One:  The  Joint  Staff  cancelled  the  AJMRR  and  JEMP  III 
projects,  GSORTS  (E)(0),  now  called  RAS  OT,  was  never  designed 
to  carry  the  full  load  of  the  original  RAS  concept  as  outlined 
in  the  Business  Plan,  It  would  be  incorrect  to  apply  standards 
set  for  the  holistic  RAS  concept  to  RAS  OT  as  a  stand-alone 
application. 

Point  Two:  The  RAS  Business  Plan  does  not  indicate  that  either 
of  its  two  remaining  components  were  designed  to  replace  legacy 
systems  resident  at  Combatant  Commands  or  Services.  It  is 
theoretically  possible,  but  unlikely,  for  RAS  OT  to  replace  a 
Service  tool.  Service  tools  query  their  respective  SORTS 
databases  that  are  of  greater  breadth  than  GSORTS.  Although  the 
GSORTS  database  is  large,  it  is  not  a  complete  representation  of 
Service  SORTS  databases.  Therefore,  a  tool  designed  to  query 
GSORTS  data  will  not  necessarily  cover  the  full  range  of  a 
Service  SORTS  database. 


6 


65 


Point  Three:  According  to  the  RAS  Business  Plan,  GSORTS{E)(I) 
was  designed  '^to  Replace  .service  unique  input  tools  so  that 
there  is  a  singly  tool  that  can  be  centrally  modified  as  new 
requirements  ate  added. (RAS  Business  Plan;  pg.  III-2) .  Only 
two  of  four  Services  intend  to  adopt  the  Input  Tool.  DISA  is 
developing  Input  Tools  for  the  Air  Force  and  Marine  Corps.  The 
Army  and  Navy's  custom  input  tools  provide  Service-specific 
data.  This  data  Is  required  only  for  Title  10  responsibilities 
and  is  beyond  the  scope  of  the  Joint  SORTS  database  and, 
therefore,  RAS  Input  Tool.  It  is  not  within  DISA's  purview  to 
require  the  Army  or  Navy  to  adopt  the  RAS  Input  Tool. 

Point  Four:  The  description  of  RAS  OT  from  the  DOD  IG  draft 
report  (page  3)  is  accurate  and  captures  the  true  intent  of  the 
tool.  However,  that  paragraph  incorrectly  lists  the  acronym  for 
Global  Status  of  Resources  and  Training  System  as  SORTS.  The 
correct  term  is  GSORTS .  SORTS  and  GSORTS  are  not  one  in  the 
same.  The  DOD  IG  description  of  RAS  OT  does  not  state  that  RAS 
OT  will  replace  a  Service  tool.  The  RAS  Business  Plan  does  not 
say  that  GSORTS (E) (O) ,  (i.e.,  RAS  OT) ,  will  replace  Service 

tools.  RA$  OT  is  designed  to  replace  the  antiquated  GCCS 
application  known  as  GIQS  (Generalized  Interactive  Query 
System) .  RAS  OT  also  brings  additional  functionality  to  the 
Combatant  Commands  and  Services  that  does  not  currently  exist  in 
the  Readiness  Community, 


2.  ReadlMss  hmmmmmmmn't  Systen  Ou'^tit  Tool  Dovolopna't  and 
Fielding  (page  4)  . 

a.  "  DISA  and  the  Joint  Staff  have  not  provided  sufficient 
assurance  that  the  RAS  Output  Tool  will  satisfy  user 
requirements.^ 

DISA  nonconcurs.  DISA  is  currently  developing  RAS  OT  based  on  a 
total  of  734  requirements  derived  from  Combatant  Commands  and 
Services.  This  list  is  validated  and  updated  by  JS  J3  DDC30-R, 
as  the  readiness  requirements  authority,  based  on  input  from  the 
readiness  community.  These  requirements  are  contained  in  a 
Requirements  Traceability  Matrix  (RTM)  of  the  Systems 
Requirements  Specification  (SRS) .  They  are  well  documented  and 
briefed  as  necessary  to  the  Global  Coiranand  and  Control  Readiness 
Working  Group  (GCC  RWG) ,  the  Readiness  Assessment  System 
Executive  Steering  Committee  (RAS  ESC),  and  annually  to  the  GCC 
RWG  User  Review  Panel.  Current  breakdown  of  those  requirements 
is  as  follows.  The  first  global  release  of  RAS  OT  (i.e.  v4,0) 
will  include  395  of  the  734  total  requirements  for  the  tool. 
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These  reguirements  were  nominated  for  inclusion  into  v4;0 
because  they  are  comon  to  users  at  all  levels  and  are 
technologically  feasible  given  the  current  GCCS  architecture  and 
Readiness  reporting  structure.  These  requirements  are  based  on 
a  series  of  Joint  Application  IDevelopment  sessions  held  between 
mid- 1997  and  1999  with  Combatant  Command  and  Service 
representatives  and  approved  by  the  JS  J3  DDGO-R.  Each 
requirement  is  represented  by  a  test  case.  Requirement 
definition  is  within  the  responsibilities  of  the  Joint  Staff  as 
acknowledged  by  the  DOD  IG  on  page  1  of  the  draft  report ^  "The 
Joint  Staff...  as  the  project  sponsor  for  all  GCCS  applications ^ 
represents  the  needs  of  the  users.  Throughout  the  requirements 
validation  and  approval  process,  the.  Joint  Staff  identifies  and 
prioritizes  functional  requirements  that  must  he  satisfied  by 
GCCS." 

The  last  formal,  comprehensive  review  of  the  RTM  occurred  in 
June  2001  in  support  of  the  RAS  OT  Functional  Qualification 
Test.  The  results  of  the  review  were  documented  and  briefed  to 
the  RAS  ESC  in  July  2001,  The  results  of  the  test  and 
recommendations  to  the  RAS  ESC  provided  a  stable  list  of 
requirements  for  RAS  OT  v4,0.  The  current  timeline  was  derived 
from  the  engineering  effort  required  to  provide  the  capability 
defined  in  these  requirmehts.  The  timeline  also  incorporates  a 
major  reengineering  effort  to  produce  a  robust,  responsive  tool 
designed  around  user  specified  requirements. 

During  the  requirements  validation,  290  of  the  734  requirements 
were  moved  to  the  category  of  future  requirements  (i.e.  beyond 
RAS  OT  v4.0)  because  they  are  either  Service  unique  or  not 
within  the  technical  capability  of  RAS  OT  v4 . 0 .  JS  J3  DDGO-R 
has  indicated  they  will  vet  the  requirements  with  Combatant 
Commands  and  Services  during  Phase  III  as  defined  in  the  RAS 
Business  Plan.  The  RAS  Business  Plan  also  coincides  with  Phase 
III  of  the  revised  RAS  OT  schedule  as  approved  by  the  RAS  ESC  in 
July  2001. 

The  remaining  49  of  734  requirements  were  designated  as  deleted 
in  the  RTM  because  they  were  either  duplicates  of  other  valid 
requirements,  impossible  to  measure,  or  beyond  the  technological 
capability  of  foreseeable  versions  of  GSORTS  or  RAS  OT. 
Requirements  deleted  from  consideration  are  still  maintained  in 
the  RTM  for  traceability, 

b,  ^DISA  and  the  Joint  Staff  failed  to  meet  RAS  Output 
Tool  development  and  fielding  milestones.  This  condition 
occurred  because  DXSA  and  the  Joint  Staff  did  not  involve  the 
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Deleted 


Revised 


users  throughout  the  development  process ^  implement  adequate 
testing  procedures f  fully  docjment  critical  decisions,  and 
establish  effective  perfoimsnce  measurements  to  monitor 
development," 

(1)  "DJ5A  and  the  Joint  Staff  failed  to  meet  RAS 
Output  Tool  development  and  fielding  milestones," 

PISA  concurs.  The  fielding  of  RAS  OT  has  been  delayed  several 
times  as  indicated  on  page  10  of  the  IG's  report, 

(2)  "This  occurred  because  DISA  and  the  Joint  Staff 
did  not  involve  the  user  community  throughout  the 
development  process." 

DISA  nonconcurs.  As  described  earlier,  users  define  the 
requirements.  However,  the  users  have  not  been  involved  in 
early  phases  of  the  developmental  process  because  it  is  not 
appropriate.  This  includes  the  developmental  phase  of  the 
system  acquisition  process.  Users  are  essential  in  the  latter 
stages  of  developmental  testing  and  are  the  backbone  of 
operational  testing  as  user  feedback  provided  on  systems  prior 
to  fielding  decisions  is  invaluable  to  the  developmental 
process.  The  JS  J-3  DDGO-R  has  mandated  such  user  involvement. 
The  first  Functional  Users  Test  of  RAS  OT  v4.0  is  currently 
scheduled  for  June  2002. 

(3)  "DISA  and  the  Joint  Staff  failed  to  meet  RAS 
Output  Tool  development  and  fielding  milestones.  This 
condition  occurred  because  DISA  and  the  Joint  Staff 
did  not... implement  adequate  testing  procedures.,," 

DISA  nonconcurs.  The  delays  were  a  direct  result  of  system 
failures  identified  by  formal,  planned  tests.  Specifically, 
these  were  the  Functional  Qualification  Tests  conducted  in  Mar 
2001  and  again  in  June  2001. 

(4)  "PISA  and  the  Joint  Staff  failed  to  meet  RAS 
Output  Tool  development  and  fielding  milestones.  This 
occurred  because  DISA  and  the  Joint  Staff  did  not...  * 
fully  document  critical  decisions,.."’ 

DISA  nonconcurs.  Considerable  documentation  related  to  critical 
decisions  was  presented  to  the  DOD  IG  team  and  is  portrayed  in 
the  "RAS  Output  Tool  Fielding  Delays"  table  on  page  10  of  the 
draft  audit  report. 
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3.  Usttr  €)  . 

a.  "ffe  kuxy^yed  each  of  the  four  Services  and  six  of  the 
nine  Unified  Commands  to  assess  user  awareness  and  participation 
in  the  RAS  Output  Tool  requirement  identification  and 
development.  The  U,S,  European  Coimandr  the  U.S.  Pacific 
Ccamand,  the  U,S-  Special  Operations  Commandf  and  the  Air  Force 
stated  they  had  limited  knowledge  of  the  RAS  Output  Tool,  In 
addition^  the  U,S,  Central  Command,  the  U*S,  Southern  Command, 
and  the  Navy  all  stated  that  they  had  no  knowledge  of  the  RAS 
Output  Tooi.* 

PISA  concurs.  At  the  time  of  the  POD  IG  survey,  Combatant 
Commands  had  had  limited,  if  any,  exposure  to  the  development 
efforts  of  RAS  OT.  In  September  2001,  based  upon  a  favorable 
lab  assessment  of  RAS  OT  and  as  a  result  of  operational  need 
arising  from  the  Global  War  on  Terrorism,  JS  j3  DDGO-R  requested 
installation  of  ah  early  release  (i.e.  engineering  drop)  of  RAS 
OT.  DISA,  with  Joint  Staff  concurrence,  has  continued  with  the  ' 
phased  installations  of  a  pre-release,  functioning  version  of 
RAS  OT  in  the  readiness  organizations  of  Combatant  Commands  and 
Services.  The  RAS  OT  has  been  delivered  to  the  following 
commands : 

JS  J3  DDGO-R 
USEUCOM 
HQ  USMC 

MARFORLANT 
MARFORRES 
HQ  USAF 
HQ  USN 
HQ  USA 
USJFCOM 
USTRANSCOM 

Installation  is  projected  at  USSOCOM,  HQ  ACC,  USAFE,  and  AFSOC 
in  the  next  few  months. 

USJFCOM  "stated  that  the  RAS  Output  Tool  did  not  provide  JRAMS 
functionality  and  that  the  application  performed  significantly 
slower  than  JRAMS. 

DISA  concurs  with  comment.  The  RAS  OT  does  contain  value-added 
JRAMS  functionality.  However,  since  RAS  OT  is  designed  to  meet 
common  Combatant  Command  and  Service  specifications,  vice  one 
specific  command,  not  every  aspect  of  JRAMS  is  included  in  RAS 
OT,  In  addition,  JRAMS  did  not  satisfy  the  Readiness 
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community's  requirements  in  1997  nor  was  it  compliant  with  the 
GCCS  architecture  or  compatible  with  GCCS  security  and 
validation  protocols. 

RAS  OT  is  slower  than  JRAMS  because  it  operates  on  a  networked 
database  vice  a  stand  alone  one,  processes  more  information,  and 
addresses  a  much  larger  collection  of  user  defined  requirements. 
Having  stated  that,  the  Joint  Staff  was  not  satisfied  with  the 
speed  of  the  RAS  OT  either.  The  fundamental  lack  of  speed  was  a 
major  driving  force  behind  DISA's  recommendation  to  the  RAS  ESC 
in  July  2001  to  reengineer  the  tool. 

b.  RAS  Business  Plan  Comments 

PISA  nonconcurs  with  the  findings  based  on  SecDef  and  Army 
comments  as  detailed  below.  The  Business  Plan  was  vetted  by 
working  groups  during  its  development  and  all  Services  provided 
comments  that  were  incorporated  in  the  final  product.  In  the 
end,  each  Service  concurred. 

(1)  SecDef  Comments.  The  POD  IG  cites  a  specific 
SecDef  " siibstantive  comment"  to  be  included  in  the  revision  of 
the  Business  Plan,  "The  key  to  the  utility  and  effectiveness  of 
the  RAS  is  the  involvement  of  the  CINCs„.and  other  users  in  the 
development  process.  Early  involvement  will  improve 
understanding  of  the  potential  of  RAS  and  will  lead  to  the 
identification  of  new  requirements."  The  POP  IG  uses  this  quote 
out  of  context.  The  above  quote  was  specifically  directed  to  be 
added  to  Chapter  IV  at  the  bottom  of  page  IV^2  of  the  RAS 
Business  Plan.  This  section  addresses  Phase  III,  "Integration 
of  New  Indicators,  Defined."  This  chapter  is  dedicated  to  a 
phase  of  RAS  development  that  has  not  yet  begun.  According  to 
the  Business  Plan,  "Phase  III  involves  the  future  planned 
integration  of  selected  portions  of  the  Joint  Total  Asset 
Visibility  (JTAV)  system.  Joint  Personnel  Asset  Visibility 
(JPAV)  system,  and  Defense  Integrated  Military  Human  Resources 
System  (DIMHRS)  leading  indicators  into  RAS,"  This  capability 
is  too  far  down  the  road  to  conceptualize  yet,  let  alone  make 
deliberate  plans.  The  SecDef  comment  used  by  the  POD  IG  to 
substantiate  their  case  for  requirements  development  does  not 
apply  to  the  phase  of  RAS  OT  development  currently  in  execution 
and  addressed  by  this  audit. 

(2)  Army  Comments.  The  DOD  IG  quote  of  the  Army 
comment  to  the  RAS  Business  plan  is  inaccurately  represented. 

The  DOD  IG  report  quotes  the  Army  comment  as  follows,  "However, 
Army  officials  did  not  believe  that  DISA  and  the  Joint  Staff  had 
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performed  an  adequate  mission  needs  analysis  or  validated  the 
requirement  for  a  readiness  assessment  application  as 
comprehensive  as  The  Amy  comment  did  not  mention 

GSORTS (E)(0)  (now  called  RAS  OT) ,  but  was  instead  a  comment 
directed  at  the  full  RAS  Business  Plan.  The  true  intent  of  the 
Army’s  comment  can  be  detemined  by  examination  of  the  two 
paragraphs  following  the  one  quoted  in  the  Draft  report.  The 
Army  had  no  worry  about  the  tool  meeting  their  needs.  They  were 
worried  that  the  RAS  Business  Plan  was  too  invasive,  perhaps 
intruding  on  the  Services'  Title  10  responsibilities.  The 
subsequent  paragraphs  from  the  Army  statement  read  as  such: 

"The  recurring  concepts  supporting  RAS  are  "commanders  at 
all  levels"  and  "near  real  time"  readiness  information s  Army 
believes  a  more  thorough  mission  needs  analysis  is  needed  to 
determine  more  precisely  the  information  timeliness  needs  at 
each  level  of  command,  and  to  target  that  information  to  the 
needs  of  the  recipient.  For  instance,  policy  makers  do  not  need 
minute  details  or  24~hour  readiness  updates  to  carry  out  policy¬ 
making  responsibilities .  Additionally,  in  the  world  of 
deliberate  planning,  current  readiness  data  is  not  useful. 

Force  sourcing  decisions  drive  resourcing,  which  ultimately 
determines  readiness.  Only  in  the  sourcing  of  contingency 
operation  force  requirements  is  current  readiness  a  factor.... 

RAS  seems  to  be  driven  more  by  the  fact  that  the  technology 
to  achieve  a  directed  telescope  to  the  forward  foxhole  will  soon 
be  available  than  by  any  validated  need.  Army  acknowledges  that 
RAS  consists  of  a  grouping  of  systems  and  software/  which  are 
already  in  some  state  of  development  or  fielding,  many  of  which 
are  already  funded  and  are  intended  to  be  resident  in  the 
GCCS/GCSS  suites.  However,  Army  believes  that  the  true  cost  of 
a  comprehensive  RAS  has  not  been  determined,  finally.  Army  is 
concerned  that  the  potential  for  micromanagement  inherent  in  RAS 
represents  an  intentional  or  unintentional  intrusion  into  the 
Services'  title  X  responsibilities  to  provide  ready  forces  to 
the  combatant  ccmmanders." 

The  DOD  IG  snapshots  of  RAS  Business  Plan  comments  from  the 
Navy,  Air  Force,  and  Marine  Corps  are  more  accurate  but  still 
represent  the  RAS  Business  Plan  in  its  entirety,  not  necessarily 
the  GSORTS (E) (0),  now  known  as  RAS  OT. 


4 .  Testing  (Page  8>  . 
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a*  '^DISA  and  the  Joint  Staff  failed  to  iii^plement  adequate 
testing  procedures  to  address  user  concerns  about  the 
opera tional  capaJbi  1  i of  the  appli ca ti on ,  Wi th  the  except i on 
of  the  U,S.  Joint  Forces  Command^  the^  tinified  Commands  and  the 
Services  had  not  participated  in  RAS  Output  Tool  testing. 

PISA  nonconcurs.  As  stated  in  comment  2;b.  (2)  above,  it  is 
inappropriate  for  users  to  participate  too  early  in  the 
developmental  testing  process.  If  users  are  involved  too  early 
in  the  testing  process,  confidence  in  the  final  system  can  be 
irretrievably  damaged.  Users  not  eminently  familiar  with 
systems  development  methodology  would  provide  conflicting  or 
confusing  feedback.  Such  input  causes  the  development  team  to 
resolve  the  conflict,  and  then  seek  to  address  things  that 
normally  would  be  addressed  in  the  course  of  developmental 
testing  anyway.  This  would  not  enhance  the  process.  However, 
once  the  product  reaches  the  latter  stages  of  developmental 
testing  and  clearly  the  operational  testing  phase,  user  input  is 
essential.  The  JS  J3  DDGO-R  mandate  insures  that  functional 
user  testing  and  operational  testing  will  rely  heavily  upon 
representatives  from  the  user  community.  A  Functional  User  Test 
is  scheduled  for  June  2002. 

Regarding  US JFCOM' s  statements:  USJFCOM  was  invited  to  view  the  ■ 
work  in  progress  because  PISA  wanted  feedback  on  the  JRAMS 
functionality  being  incorporated  into  RAS  OT.  This  was  a 
demonstration,  not  a  test.  DiSA  received  valuable  feedback  from 
USJFCOM  and  JS  J3  DPGO^R  validated  eight  new  requirements  to  be 
included  in  RAS  OT  v4.0. 

b.  "Since  the  time  of  the  preliminary  requirements 
definition r  DISA  has  provided  limited  hands-on  testing 
experience,  which  has  diminished  the  potential  that  the  users 
will  gain  the  confidence  that  the  RAS  Output  Tool  will  satisfy 
their  needs." 

DISA  nonconcurs.  As  noted  above,  including  users  during  early 
stages  of  development  is  not  a  productive  use  of  either 
developer  or  user  resources.  The  USJFCOM  comments  reflect  the 
application  of  standards  for  mature  systems,  like  JRAMS,  against 
systems  in  development  such  as  RAS  OT,  The  involvement  of  users 
too  early  in  the  process  usually  results  in  a  loss  of  confidence 
in  the  system  and  its  developers .  Through  the  engineering 
release  of  the  RAS  OT  as  described  in  3. a.  above,  numerous  users 
have  had  the  opportunity  to  provide  both  formal  and  informal 
feedback  for  further  refinement  of  functionality  against 
validated  requirements  based  on  hands-on  experience.  In 
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addition,  this  has  also  allowed  theses  users  to  integrate  the 
tool  in  support  of  real  world  current  operations.  The 
Functional  User  Test  scheduled  for  June  2002  will  certainly  . 
.involve  the  user  community.  Inclusion  of  users  will  also  occur 
during  the  operational  testing  phase  in  the  project’s  schedule. 

c.  "The  Joint  Staff  and  DISA  did  not  maintain  adequate 
documentation  of  critical  decisions  made  during  RAS  Output  Tool 
development.'^ 

DISA  nonconcurs .  Critical  decisions  have  been  documented.  The 
following  documents  are  cited  as  examples: 

•  In  Progress  Review  (IPR)  briefing  slides  (24  Nov  1999). 

DISA  presented  a  tactical  way  ahead  for  RAS,  assessed  the 
risk  associated  with  this  course  of  action,  and  addressed 
the  Joint  Staff  concerns. 

•  RAS  IPR  meeting  minutes  (6  Apr  2000).  The  minutes 
addressed  the  issues  of  RAS  OT  shortcomings  and  DISA  D6 
response  to  those  issues. 

•  RAS  IPR  meeting  minutes  (22  Nov  2000) .  The  minutes 
document  the  concerns  of  the  Joint  Staff  and  due-outs 
resulting  from  those  concerns.  One  of  the  items  led  to  the 
formation  of  the  RAS  ESC  and  its  supporting  RAS  Action 
Officer's  Working  Group  (RAS  AWG) . 

•  RAS  ESC  briefing  slides  (23  Feb  2001} .  This  RAS  ESC 
addressed  the  10-month  slip  briefed  at  the  22  Nov  2000  IPR 
and  the  decision  to  perform  a  JRAMS-E  versus  a  RAS  OT 
assessment. 

•  RAS  ESC  meeting  minutes  (20  Apr  2001) .  RAS  ESC  addressed 
the  current  schedule,  short  falls  identified  by  the 
Functional  Qualification  Test  (21  Mar  2001),  and  a  course 
of  action  (COA)  designed  to  mitigate  those  issues. 

•  RAS  ESC  meeting  minutes  (25  Jul  2001).  RAS  ESC  addressed 
the  progress  made  as  of  the  Functional  Qualification  Test 
(22  Jun  2001) .  RAS  ESC  prescribed  the  way  ahead  to  address 
the  remaining  deficiencies,  acknowledged  DISA's  proposed 
schedule  of  events,  and  assessed  the  risk's  associated  with 
this  COA. 

There  is  also  a  volume  of  correspondence  related  to  the  RAS  OT 
(then  called  GSORTS  (E)  (0) )  statement  of  work.  The  period  of 

correspondence  is  from  January  1998  to  July  2000.  This  provides 
considerable  detail  into  the  genesis  of  issues  leading  to  RAS  OT 
shortcomings  and  missed  milestones.  There  is  a  similar  volume. 
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although  much  smaller,  for  the  current  contract  covering  RAS 
applications  in  general,  including  HAS  OT. 

It  would  be  helpful  if  the  DOD  IG  would  identify  which  critical 
decisions  DISA  did  not  document,  and  applicable  references 
delineating  how  such  decisions  should  have  been  documented. 

d.  "In  addition,  an  adeguate  mechanism  for  measuring 
performance  was  not  established. ^ 

DISA  concurs  with  comment.  In  a  November  2000  GCCS  Program 
Management  Review,  OASD  (C3I)  recognized  the  need  for  a  GCCS 
Operational  Requirements  Document  (ORD)  and  tasked  the  Joint 
Staff  to  develop  an  GCCS  ORD  and  submit  it  to  the  JROC  for 
approval  in  FY02.  The  GCCS  ORD  will  identify  objective  and 
thresholds  for  operational  requirements,  as  well  as  key 
performance  parameters,  for  the  GCCS  system  which  will  include 
RAS  OT  as  a  mission  application  of  GCCS. 

e.  "In  addition,  delays  in  fielding  made  it  necessary  for 
the  Unified  Commands  and  Services  to  develop  new  applications  or 
modify  existing  applications  that  would  satisfy  their  unique 
requirements  for  performing  readiness  analysis." 

DISA: nonconcurs.  This  assertion  draws  an  incorrect  corollary 
between  RAS  OT  delays  and  development  of  Combatant  Command  or 
Service  tools.  The  systems  within  RAS  designed  to  perform 
readiness  analysis,  AJMRR  and  GSORTS (E) {0} ,  were  never  intended 
to  replace  Combatant  Command  or  Service  specific  tools.  The 
GSORTS (E)(1)  was  designed  to  replace  selected  Service  input 
tools,  but  the  input  tool  is  not  an  application  for  performing 
readiness  analysis. 

In  addition.  Combatant  Command  and  Service  readiness 
applications  predate  the  first  slip  in  the  RAS  development 
schedule  identified  to  the  RAS  ESC  in  November  1999.  USJFCOM's 
JRAMS  began  development  in  1995  and  JRAMS  Classic  was  fielded  in 
1997,  The  Army  system,  ARMS,  has  been  in  use  since  1989,  The 
Navy's  TRMS  predates  1999  as  does  the  Marine  Corps'  GOMERS, 

DISA  is  not  aware  of  any  Combatant  Command  or  Service  that  has 
based  a  readiness  development  decision  on  RAS  OT  schedule 
delays , 


5.  RXS  Out^t  Tool  Coats. 
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”DISA  has  spent  $1.3  million  directly  allocatable  to  the  RAS 
Output  Tool.  DISA  has  also  spent  an  unknown  portion  of  an 
additional  $9.2  million,  classified  as  readiness  applications 
support,  for  the  RAS  Output  Tool.” 

DISA  concurs  with  coinment.  RAS  OT  expenditures  total 
approximately  $6.1  million  between  01  January  1998  and  31  March 
2002.  Approximately  $1.72  million  of  this  total  resulted  from 
1998  development  efforts  by  SAIC.  Initial  effort  was  under  the 
Defense  Advanced  Research  Project  Agency's  (DARPA)  management. 
DISA  took  over  the  project  in  April  1998  using  that  developer. 
Subsequently,  DISA  merged  the  RAS  effort  with  ongoing  projects. 
The  new  contract  effort  initiated  in  January  1999  with  AB  Floyd, 
Inc.  Approximately  $4,3  million  has  been  expended  on  RAS  OT 
development  between  11  January  1999  and  31  March  2002. 

6.  DOD  IG  Conclusion,  (page  14) 

"  The  DoD  has  spent  more  than  5  years  and  millions  of  dollars 
attempting  to  develop  and  field  a  readiness  analysis  application 
that  will  not  be  available  for  at  least  another  18  months.  The 
Joint  St^ff  did  not  adequately  document  the  need  for  the  RAS 
Output  Tool.  Also,  DISA  does  not  appear  to  have  thoroughly 
investigated  the  costs  and  feasibility  of  Yielding  the  RAS 
Output  Tool  and  does  not  appear  to  be  testing  operational 
performance  regulrements  that  agree  with  user  requirements . 
Consequently,  the  program  has  encountered  numerous  fielding 
delays  and  DISA  has  started  a  redesign  effort  to  make  the 
application  operational.  Furthermore,  the  RAS  Output  Tool  may 
not  satisfy  readiness  analysis  requirements  for  the  user.  In 
addition,  DoD  did  not  establish  performance  measures  that  would 
facilitate  effective  program  management  and  oversight  and  help 
ensure  that  DISA  fields  a  readiness  analysis  application." 

DISA  concurs  with  comment.  DISA  has  taken  a  number  of  steps 
over  the  last  18  months  to  achieve  stable  development, 
integration,  testing,  and  fielding  plans  for  RAS  OT.  Those 
steps  are  described  in  the  responses  to  findings  in  this  report, 
as  shown  above.  The  DOD  IG  Audit  Report  does  not  recognize  the 
significant  progress  that  has  been  made  over  the  last  18  months. 
Unfortunately,  the  report  focuses  on  the  issues  from  the 
previous  three  years.  An  early  release  of  RAS  OT  is  now 
installed  at  a  number  of  Combatant  Command  and  Service  locations 
and  the  users  at  those  sites  are  extremely  positive  in  their 
assessment  of  the  operational  usefulness  of  the  tool.  As  DISA 
completes  the  GCCS  Block  IV  Implementation  Plan,  cost. 
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feasibility,  operational  performance  requirements,  and 
performance  measures  will  be  addressed  for  major  Block  IV 
capabilities,  iiicluding  E%AS  OT- 
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